win10下运行hadoop程序报错

在win10的eclipse上运行一段代码需要用到hadoop。一开始报错:Could not locate executable null\bin\winutils.exe in the Hadoop binaries
按照指导下载了winutils.exe 并且配置了环境变量。再运行程序又报错:
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.ClientWrapper.(ClientWrapper.scala:171)
at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:162)
at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:160)
at org.apache.spark.sql.hive.HiveContext.(HiveContext.scala:167)
at org.apache.spark.sql.CarbonContext.(CarbonContext.scala:34)
at org.carbondata.examples.util.InitForExamples$.createCarbonContext(InitForExamples.scala:42)
at org.carbondata.examples.CarbonExample$.main(CarbonExample.scala:26)
at org.carbondata.examples.CarbonExample.main(CarbonExample.scala)
Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: ---------
at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:612)
at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:554)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
... 8 more

按照指导在win10 cmd窗口后运行 winutils.exe chmod 777 /tmp/hive
报如下错误:
ChangeFileModeByMask error (3): ???????????
哪位大神现身救助

3个回答

winutils.exe chmod 777 C:/tmp/hive

权限问题,应该是命令有误,建议设置一下cmd的编码显示问题,看看究竟“???????????”报的什么错,再根据情况修改

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
win10下编译hadoop eclipse plugin报错
win10下编译hadoop eclipse plugin报错,请求各位大佬帮忙看一下 ``` D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin>ant jar -Dversion=2.8.3 -Declipse.home=C:\Users\Daybr\eclipse\java-neon\eclipse -Dhadoop.home=D:\hadoop-2.8.3\hadoop-2.8.3 Buildfile: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml check-contrib: init: [echo] contrib: eclipse-plugin init-contrib: ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\ivy\ivysettings.xml ivy-resolve-common: ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\ivy\ivysettings.xml compile: [echo] contrib: eclipse-plugin [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 45 source files to D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\build\contrib\eclipse-plugin\classes [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\Activator.java:22: 错误: 程序包org.eclipse.ui.plugin不存在 [javac] import org.eclipse.ui.plugin.AbstractUIPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\Activator.java:28: 错误: 找不到符号 [javac] public class Activator extends AbstractUIPlugin { [javac] ^ [javac] 符号: 类 AbstractUIPlugin [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ErrorMessageDialog.java:22: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Display; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:21: 错误: 程序包org.eclipse.debug.ui不存在 [javac] import org.eclipse.debug.ui.IDebugUIConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:22: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.JavaUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:23: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IFolderLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:24: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IPageLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:25: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IPerspectiveFactory; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:26: 错误: 程序包org.eclipse.ui.console不存在 [javac] import org.eclipse.ui.console.IConsoleConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:34: 错误: 找不到符号 [javac] public class HadoopPerspectiveFactory implements IPerspectiveFactory { [javac] ^ [javac] 符号: 类 IPerspectiveFactory [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:36: 错误: 找不到符号 [javac] public void createInitialLayout(IPageLayout layout) { [javac] ^ [javac] 符号: 类 IPageLayout [javac] 位置: 类 HadoopPerspectiveFactory [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:25: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.FileLocator; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:26: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:28: 错误: 程序包org.eclipse.swt.graphics不存在 [javac] import org.eclipse.swt.graphics.Image; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:29: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.ISharedImages; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:30: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.PlatformUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:31: 错误: 程序包org.eclipse.ui.plugin不存在 [javac] import org.eclipse.ui.plugin.AbstractUIPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:46: 错误: 找不到符号 [javac] private ISharedImages sharedImages = [javac] ^ [javac] 符号: 类 ISharedImages [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:70: 错误: 找不到符号 [javac] public static Image getImage(String name) { [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:95: 错误: 找不到符号 [javac] private Map<String, Image> imageMap = new HashMap<String, Image>(); [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:154: 错误: 找不到符号 [javac] private Image getImageByName(String name) { [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:29: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:30: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProjectNature; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:31: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:32: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.NullProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:33: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:34: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.QualifiedName; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:35: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IClasspathEntry; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:36: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IJavaProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:37: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.JavaCore; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:44: 错误: 找不到符号 [javac] public class MapReduceNature implements IProjectNature { [javac] ^ [javac] 符号: 类 IProjectNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:48: 错误: 找不到符号 [javac] private IProject project; [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:56: 错误: 找不到符号 [javac] public void configure() throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:130: 错误: 找不到符号 [javac] public void deconfigure() throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:137: 错误: 找不到符号 [javac] public IProject getProject() { [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:145: 错误: 找不到符号 [javac] public void setProject(IProject project) { [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:21: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IFile; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:22: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:23: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:24: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IJavaElement; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:25: 错误: 程序包org.eclipse.jdt.internal.ui.wizards不存在 [javac] import org.eclipse.jdt.internal.ui.wizards.NewElementWizard; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:28: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.INewWizard; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:29: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IWorkbench; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:36: 错误: 找不到符号 [javac] public class NewDriverWizard extends NewElementWizard implements INewWizard, [javac] ^ [javac] 符号: 类 NewElementWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:36: 错误: 找不到符号 [javac] public class NewDriverWizard extends NewElementWizard implements INewWizard, [javac] ^ [javac] 符号: 类 INewWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:23: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:24: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.FileLocator; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:25: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:26: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IStatus; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:27: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:28: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IType; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:29: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.JavaModelException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:30: 错误: 程序包org.eclipse.jdt.core.search不存在 [javac] import org.eclipse.jdt.core.search.SearchEngine; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:31: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.IJavaElementSearchConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:32: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.JavaUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:33: 错误: 程序包org.eclipse.jdt.ui.wizards不存在 [javac] import org.eclipse.jdt.ui.wizards.NewTypeWizardPage; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:38: 错误: 程序包org.eclipse.swt不存在 [javac] import org.eclipse.swt.SWT; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:39: 错误: 程序包org.eclipse.swt.layout不存在 [javac] import org.eclipse.swt.layout.GridData; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:40: 错误: 程序包org.eclipse.swt.layout不存在 [javac] import org.eclipse.swt.layout.GridLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:41: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Button; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:42: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Composite; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:43: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Event; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:44: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Label; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:45: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Listener; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:46: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Text; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:47: 错误: 程序包org.eclipse.ui.dialogs不存在 [javac] import org.eclipse.ui.dialogs.SelectionDialog; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:54: 错误: 找不到符号 [javac] public class NewDriverWizardPage extends NewTypeWizardPage { [javac] ^ [javac] 符号: 类 NewTypeWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:43: 错误: 找不到符号 [javac] public void run(IProgressMonitor monitor) { [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:60: 错误: 找不到符号 [javac] public void init(IWorkbench workbench, IStructuredSelection selection) { [javac] ^ [javac] 符号: 类 IWorkbench [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:90: 错误: 找不到符号 [javac] protected void finishPage(IProgressMonitor monitor) [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:91: 错误: 找不到符号 [javac] throws InterruptedException, CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:96: 错误: 找不到符号 [javac] public IJavaElement getCreatedElement() { [javac] ^ [javac] 符号: 类 IJavaElement [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:55: 错误: 找不到符号 [javac] private Button isCreateMapMethod; [javac] ^ [javac] 符号: 类 Button [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:57: 错误: 找不到符号 [javac] private Text reducerText; [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:59: 错误: 找不到符号 [javac] private Text mapperText; [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:85: 错误: 找不到符号 [javac] public void createType(IProgressMonitor monitor) throws CoreException, [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:85: 错误: 找不到符号 [javac] public void createType(IProgressMonitor monitor) throws CoreException, [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:91: 错误: 找不到符号 [javac] protected void createTypeMembers(final IType newType, ImportsManager imports, [javac] ^ [javac] 符号: 类 IType [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:91: 错误: 找不到符号 [javac] protected void createTypeMembers(final IType newType, ImportsManager imports, [javac] ^ [javac] 符号: 类 ImportsManager [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:92: 错误: 找不到符号 [javac] final IProgressMonitor monitor) throws CoreException { [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:92: 错误: 找不到符号 [javac] final IProgressMonitor monitor) throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:145: 错误: 找不到符号 [javac] public void createControl(Composite parent) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:199: 错误: 找不到符号 [javac] private void createMapperControls(Composite composite) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:204: 错误: 找不到符号 [javac] private void createReducerControls(Composite composite) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:209: 错误: 找不到符号 [javac] private Text createBrowseClassControl(final Composite composite, [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:209: 错误: 找不到符号 [javac] private Text createBrowseClassControl(final Composite composite, [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:29: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:30: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProjectDescription; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:31: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.ResourcesPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:32: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:33: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IConfigurationElement; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:34: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IExecutableExtension; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:36: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.NullProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:37: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:38: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.QualifiedName; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:39: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.SubProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:40: 错误: 程序包org.eclipse.jdt.ui.wizards不存在 [javac] import org.eclipse.jdt.ui.wizards.NewJavaProjectWizardPage; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:49: 错误: 程序包org.eclipse.swt不存在 [javac] import org.eclipse.swt.SWT; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:50: 错误: 程序包org.eclipse.swt.events不存在 [javac] import org.eclipse.swt.events.SelectionEvent; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:51: 错误: 程序包org.eclipse.swt.events不存在 [javac] import org.eclipse.swt.events.SelectionListener; [javac] ^ [javac] 注: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\dfs\DFSFolder.java使用或覆盖了已过时的 API。 [javac] 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 [javac] 注: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\actions\DFSActionImpl.java使用了未经检查或不安全的操作。 [javac] 注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。 [javac] 注: 某些消息已经过简化; 请使用 -Xdiags:verbose 重新编译以获得完整输出 [javac] 100 个错误 BUILD FAILED D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml:76: Compile failed; see the compiler error output for details. Total time: 4 seconds ```
Myeclipse集成Hadoop运行程序报错: Input path does not exist
各位大神,求问~~ 我在win7-VMware-Ubuntu下搭建了Hadoop伪分布环境,在win7下安装了Myeclipse,并集成Hadoop,运行程序时报如下错: ![图片说明](https://img-ask.csdn.net/upload/201511/18/1447777056_5998.png) 但hdfs中有_LPH.txt这个文档: ![图片说明](https://img-ask.csdn.net/upload/201511/18/1447777359_756547.png) 问题可能出在哪里?叩谢~~~
window下连接hadoop集群报错
window下连接hadoop集群报错,已经把hadoop.dll放在window下的hadoop的bin目录了,system32也放了,还是无效,请问怎么办??![图片说明](https://img-ask.csdn.net/upload/201508/31/1441030623_267379.png)
hadoop jar 运行hadoop自带wordcount报错
安装完hadoop-2.7.3之后,想试试其自带的wordcount例子,但是一直报下面的错,空间不足。。 但是我都还没开始用,怎么会出这种问题。。我是单机上运行的伪分布式,其他的都能正常运行,hdfs也是正常的,能在其中导入文件啥的,但是就是不能用hadoop jar命名运行![图片说明](https://img-ask.csdn.net/upload/201706/05/1496656412_150716.png)
新手,hadoop上运行wordcount程序报错
运行的环境是:Ubuntu14.04+hadoop2.6.1 用的是virtualBox虚拟机,然后安装了一个master和三个slave节点 hadoop是可以成功启动的,没有任何问题 在Ubuntu安装了eclipse,用java写了word count的程序,源码如下: ``` package wordcount; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; /** * @author * @version 创建时间:2017年9月9日 上午8:50:51 类说明 */ public class Wordcount { public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException { StringTokenizer line = new StringTokenizer(value.toString()); while (line.hasMoreTokens()) { word.set(line.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable result = new IntWritable(); protected void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable obj : values) { sum += obj.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(Wordcount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("hdfs://master:9000/user/hduser/demo/test.txt")); FileOutputFormat.setOutputPath(job, new Path("hdfs://master:9000/user/hduser/demo/wordcount")); //FileInputFormat.addInputPath(job, new Path(args[0])); //FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } ``` 启动hadoop后,在eclipse中直接运行上面的程序,运行成功,生成了wordcount文件夹,里面有_SUCCESS文件,也有统计的结果文件 然后我想把程序打包成jar文件来运行,先把上面程序中的: ``` FileInputFormat.addInputPath(job, new Path("hdfs://master:9000/user/hduser/demo/test.txt")); FileOutputFormat.setOutputPath(job, new Path("hdfs://master:9000/user/hduser/demo/wordcount")); ``` 改成如下: ``` FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); ``` 就是通过终端输入这两个参数 用eclipse的export打包成jar文件,后在终端输入: ``` hadoop jar wordcount.jar wordcount.Wordcount hdfs://master:9000/user/hduser/demo/test.txt hdfs://master:9000/user/hduser/demo/wordcount ``` 运行就报错了,报错情况如下: ``` 17/09/09 11:18:53 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.56.100:8050 17/09/09 11:18:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 17/09/09 11:18:55 INFO input.FileInputFormat: Total input paths to process : 1 17/09/09 11:18:55 INFO mapreduce.JobSubmitter: number of splits:1 17/09/09 11:18:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504926710828_0001 17/09/09 11:18:56 INFO impl.YarnClientImpl: Submitted application application_1504926710828_0001 17/09/09 11:18:56 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504926710828_0001/ 17/09/09 11:18:56 INFO mapreduce.Job: Running job: job_1504926710828_0001 17/09/09 11:19:14 INFO mapreduce.Job: Job job_1504926710828_0001 running in uber mode : false 17/09/09 11:19:14 INFO mapreduce.Job: map 0% reduce 0% 17/09/09 11:19:14 INFO mapreduce.Job: Job job_1504926710828_0001 failed with state FAILED due to: Application application_1504926710828_0001 failed 2 times due to AM Container for appattempt_1504926710828_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://master:8088/proxy/application_1504926710828_0001/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1504926710828_0001_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 17/09/09 11:19:14 INFO mapreduce.Job: Counters: 0 ``` 去查了下日志文件, ``` 2017-09-09 11:18:55,869 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job.xml is closed by DFSClient_NONMAPREDUCE_-1306163227_1 2017-09-09 11:18:59,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-09-09 11:18:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). 2017-09-09 11:19:12,241 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.56.102:53610 Call#7 Retry#0: java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist 2017-09-09 11:19:12,293 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.56.102:53610 Call#8 Retry#0: java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist 2017-09-09 11:19:29,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-09-09 11:19:29,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.56.100 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 29 2017-09-09 11:19:42,635 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 6 Number of transactions batched in Syncs: 0 Number of syncs: 27 SyncTimes(ms): 545 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 6 Number of transactions batched in Syncs: 0 Number of syncs: 28 SyncTimes(ms): 613 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /usr/local/hadoop/hadoop_data/hdfs/namenode/current/edits_inprogress_0000000000000000029 -> /usr/local/hadoop/hadoop_data/hdfs/namenode/current/edits_0000000000000000029-0000000000000000068 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 69 2017-09-09 11:19:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-09-09 11:19:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:20:29,504 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-09-09 11:20:29,504 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.56.100 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 69 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 24 2017-09-09 11:20:42,791 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 56 ``` 在这里面报了一个错误: ``` java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist ``` 新手,不知道怎么办了
在eclipse运行hadoop mapreduce例子报错
在终端运行hadoop带的例子正常,hadoop节点正常,错误如下 17/09/05 20:20:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/05 20:20:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 17/09/05 20:20:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread "main" java.net.ConnectException: Call From master/192.168.1.110 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1479) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at mapreduce.Temperature.main(Temperature.java:202) Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at org.apache.hadoop.ipc.Client.call(Client.java:1451) ... 28 more
hadoop连接mongo的程序,打包成jar后,hadoop下运行报错NoClassDefFoundError
报错信息![](https://img-ask.csdn.net/upload/201910/09/1570610725_197608.png) ![图片说明](https://img-ask.csdn.net/upload/201910/09/1570610769_40828.png) 代码![图片说明](https://img-ask.csdn.net/upload/201910/09/1570610818_23375.png) eclipse下可以正常运行
Hadoop安装插件运行MapReduce报错
我的虚拟机搭建了HDFS一个集群完全分布式,在win7系统下安装Eclipse插件,运行MapReduce程序报错,好像是连接不上,求大神帮忙![图片](https://img-ask.csdn.net/upload/201608/08/1470613451_63912.jpg)![图片](https://img-ask.csdn.net/upload/201608/08/1470613343_436152.jpg)
win7 下运行hadoop的workcount卡在了map0% reduce0% 求大神指教
刚刚安装好Hadoop 就想用自带的WORDCOUNT这个程序检验一下好不好使就卡主了 求大神帮忙 愁了一个星期了 拜托拜托![图片说明](https://img-ask.csdn.net/upload/201511/15/1447592947_812710.png)
hadoop hdfs 挂载报错
hadoop hdfs挂载在linux上市报错fuse-dfs didn't recognize /hadoop/hdfs,-2,然后打开/hadoop/hdfs时报输入输出目录错误
win7 下运行hadoop wordcount 出现下面问题是为啥啊 新人求教
![图片说明](https://img-ask.csdn.net/upload/201511/15/1447590728_655101.png)期各位大神指教 我这纠结一个多星期了 拜托拜托
Hadoop 2.2运行wordcount报错
hadoop 2.2 + jdk1.7 运行wordcount例子 hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount /word /ws 报错: org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1449733659077_0001_m_000000_0: Error: java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to org.apache.hadoop.mapred.InputSplit 请各位高手指点
win7下Eclipse开发Hadoop应用程序环境搭建
这需要装cygwin么?然后我搭建的用不了![图片说明](https://img-ask.csdn.net/upload/201511/24/1448363975_932606.png)![图片说明](https://img-ask.csdn.net/upload/201511/24/1448364029_470310.png)
hadoop mapreduce报错
java.lang.RuntimeException: Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy31.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy32.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448) Job Submission failed with exception 'java.lang.RuntimeException(Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) )' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
hadoop datanode日志报错
2016-04-10 11:35:08,998 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService java.io.EOFException: End of File Exception between local host is: "master/10.13.6.186"; destination host is: "master":9000; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
Hadoop Distcp报错 队列问题
``` sudo -uxiaosi hadoop distcp hdfs:///user/xiaosi/tmp/data_group/histories/day=20161116 hdfs:///user/xiaosi/data_group/histories ``` 报错: ``` 17/01/17 19:18:46 ERROR security.UserGroupInformation: PriviledgedActionException as:xiaosi (auth:SIMPLE) cause:java.io.IOException: Failed to run job : User xiaosi cannot submit applications to queue root.default 17/01/17 19:18:46 ERROR tools.DistCp: Exception encountered java.io.IOException: Failed to run job : User xiaosi cannot submit applications to queue root.default at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:299) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153) at org.apache.hadoop.tools.DistCp.run(DistCp.java:118) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:375) ```
hadoop单词统计报错Job job_1581768459583_0001 failed
3个节点hadoop01、hadoop02、hadoop03 hadoop01是主节点 hadoop01、hadoop02、hadoop03是从节点,目前集群已搭建好,jps查看三个节点运行都很正常,而且UI也能正常显示,但是使用hadoop自带的hadoop-mapreduce-examples-2.7.4.jar的wordcount进行单词统计时报错如下,请高人指点,看不懂呀: ```[root@hadoop01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.4.jar wordcount /wordcount/input /wordcount/output 20/02/15 20:14:25 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.233.132:8032 20/02/15 20:14:27 INFO input.FileInputFormat: Total input paths to process : 1 20/02/15 20:14:27 INFO mapreduce.JobSubmitter: number of splits:1 20/02/15 20:14:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1581768459583_0001 20/02/15 20:14:28 INFO impl.YarnClientImpl: Submitted application application_1581768459583_0001 20/02/15 20:14:28 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1581768459583_0001/ 20/02/15 20:14:28 INFO mapreduce.Job: Running job: job_1581768459583_0001 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 running in uber mode : false 20/02/15 20:15:38 INFO mapreduce.Job: map 0% reduce 0% 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 failed with state FAILED due to: Application application_1581768459583_0001 failed 2 times due to Error launching appattempt_1581768459583_0001_000002. Got exception: java.io.IOException: Failed on local exception: java.io.IOException: java.io.IOException: Connection reset by peer; Host Details : local host is: "hadoop01.com/79.124.78.101"; destination host is: "79.124.78.101":43276; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy83.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy84.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.io.IOException: Connection reset by peer at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:651) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 16 more Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726) ... 19 more . Failing the application. 20/02/15 20:15:38 INFO mapreduce.Job: Counters: 0 ```
hadoop执行wordcount报错
WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 17/05/03 02:35:31 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 17/05/03 02:35:31 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null 17/05/03 02:35:31 WARN hdfs.DFSClient: Could not get block locations. Source file "/zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar" - Aborting... 17/05/03 02:35:31 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.136.131:9000/zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001 17/05/03 02:35:31 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 17/05/03 02:35:31 ERROR hdfs.DFSClient: Failed to close file /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 报错内容是没有节点,但是用jps查看都已经正常启动,而且防火墙也关闭了,节点间通信也都正常
Win10上启动Hadoop报 Error: Could not find or load main class PC
我在Windows10上安装了Hadoop 2.9.2 在hadoop-env.cmd中修改了 set JAVA_HOME=C://PROGRA~1/java/jdk1.8.0_191 set HADOOP_PREFIX=E://hadoop-2.9.2 执行Hadoop命令的时候出现奇怪的情况 情况1: 直接执行hadoop命令: ``` E:\hadoop-2.9.2\bin>hadoop Usage: hadoop [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: fs run a generic filesystem user client version print the version jar <jar> run a jar file note: please use "yarn jar" to launch YARN applications, not this command. checknative [-a|-h] check native hadoop and compression libraries availability distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive classpath prints the class path needed to get the Hadoop jar and the required libraries credential interact with credential providers key manage keys via the KeyProvider daemonlog get/set the log level for each daemon or CLASSNAME run the class named CLASSNAME Most commands print help when invoked w/o parameters. ``` 可以正常执行命令 情况二: 执行hadoop classpath ``` E:\hadoop-2.9.2\bin>hadoop classpath E:\hadoop-2.9.2\etc\hadoop;E:\hadoop-2.9.2\share\hadoop\common\lib\*;E:\hadoop-2.9.2\share\hadoop\common\*;E:\hadoop-2.9.2\share\hadoop\hdfs;E:\hadoop-2.9.2\share\hadoop\hdfs\lib\*;E:\hadoop-2.9.2\share\hadoop\hdfs\*;E:\hadoop-2.9.2\share\hadoop\yarn;E:\hadoop-2.9.2\share\hadoop\yarn\lib\*;E:\hadoop-2.9.2\share\hadoop\yarn\*;E:\hadoop-2.9.2\share\hadoop\mapreduce\lib\*;E:\hadoop-2.9.2\share\hadoop\mapreduce\* ``` 可以正常显示所有的classpath 情况三: **执行hadoop version --出错了** ``` E:\hadoop-2.9.2\bin>hadoop version Error: Could not find or load main class PC ``` 这个就出错了 我查了一些资料,说是classpath不正确,但是我看classpath的路径确实是我安装hadoop的路径,请问各位大神有没有碰到过类似情况的?麻烦分享一下,谢谢。
终于明白阿里百度这样的大公司,为什么面试经常拿ThreadLocal考验求职者了
点击上面↑「爱开发」关注我们每晚10点,捕获技术思考和创业资源洞察什么是ThreadLocalThreadLocal是一个本地线程副本变量工具类,各个线程都拥有一份线程私有的数
程序员必须掌握的核心算法有哪些?
由于我之前一直强调数据结构以及算法学习的重要性,所以就有一些读者经常问我,数据结构与算法应该要学习到哪个程度呢?,说实话,这个问题我不知道要怎么回答你,主要取决于你想学习到哪些程度,不过针对这个问题,我稍微总结一下我学过的算法知识点,以及我觉得值得学习的算法。这些算法与数据结构的学习大多数是零散的,并没有一本把他们全部覆盖的书籍。下面是我觉得值得学习的一些算法以及数据结构,当然,我也会整理一些看过...
《奇巧淫技》系列-python!!每天早上八点自动发送天气预报邮件到QQ邮箱
此博客仅为我业余记录文章所用,发布到此,仅供网友阅读参考,如有侵权,请通知我,我会删掉。 补充 有不少读者留言说本文章没有用,因为天气预报直接打开手机就可以收到了,为何要多此一举发送到邮箱呢!!!那我在这里只能说:因为你没用,所以你没用!!! 这里主要介绍的是思路,不是天气预报!不是天气预报!!不是天气预报!!!天气预报只是用于举例。请各位不要再刚了!!! 下面是我会用到的两个场景: 每日下
面试官问我:什么是消息队列?什么场景需要他?用了会出现什么问题?
你知道的越多,你不知道的越多 点赞再看,养成习惯 GitHub上已经开源 https://github.com/JavaFamily 有一线大厂面试点脑图、个人联系方式,欢迎Star和完善 前言 消息队列在互联网技术存储方面使用如此广泛,几乎所有的后端技术面试官都要在消息队列的使用和原理方面对小伙伴们进行360°的刁难。 作为一个在互联网公司面一次拿一次Offer的面霸,打败了无数
8年经验面试官详解 Java 面试秘诀
    作者 | 胡书敏 责编 | 刘静 出品 | CSDN(ID:CSDNnews) 本人目前在一家知名外企担任架构师,而且最近八年来,在多家外企和互联网公司担任Java技术面试官,前后累计面试了有两三百位候选人。在本文里,就将结合本人的面试经验,针对Java初学者、Java初级开发和Java开发,给出若干准备简历和准备面试的建议。   Java程序员准备和投递简历的实
究竟你适不适合买Mac?
我清晰的记得,刚买的macbook pro回到家,开机后第一件事情,就是上了淘宝网,花了500元钱,找了一个上门维修电脑的师傅,上门给我装了一个windows系统。。。。。。 表砍我。。。 当时买mac的初衷,只是想要个固态硬盘的笔记本,用来运行一些复杂的扑克软件。而看了当时所有的SSD笔记本后,最终决定,还是买个好(xiong)看(da)的。 已经有好几个朋友问我mba怎么样了,所以今天尽量客观
程序员一般通过什么途径接私活?
二哥,你好,我想知道一般程序猿都如何接私活,我也想接,能告诉我一些方法吗? 上面是一个读者“烦不烦”问我的一个问题。其实不止是“烦不烦”,还有很多读者问过我类似这样的问题。 我接的私活不算多,挣到的钱也没有多少,加起来不到 20W。说实话,这个数目说出来我是有点心虚的,毕竟太少了,大家轻喷。但我想,恰好配得上“一般程序员”这个称号啊。毕竟苍蝇再小也是肉,我也算是有经验的人了。 唾弃接私活、做外
大学四年自学走来,这些珍藏的「实用工具/学习网站」我全贡献出来了
知乎高赞:文中列举了互联网一线大厂程序员都在用的工具集合,涉及面非常广,小白和老手都可以进来看看,或许有新收获。
《阿里巴巴开发手册》读书笔记-编程规约
Java编程规约命名风格 命名风格 类名使用UpperCamelCase风格 方法名,参数名,成员变量,局部变量都统一使用lowerCamelcase风格 常量命名全部大写,单词间用下划线隔开, 力求语义表达完整清楚,不要嫌名字长 ...
Python爬虫爬取淘宝,京东商品信息
小编是一个理科生,不善长说一些废话。简单介绍下原理然后直接上代码。 使用的工具(Python+pycharm2019.3+selenium+xpath+chromedriver)其中要使用pycharm也可以私聊我selenium是一个框架可以通过pip下载 pip install selenium -i https://pypi.tuna.tsinghua.edu.cn/simple/ 
阿里程序员写了一个新手都写不出的低级bug,被骂惨了。
你知道的越多,你不知道的越多 点赞再看,养成习惯 本文 GitHub https://github.com/JavaFamily 已收录,有一线大厂面试点思维导图,也整理了很多我的文档,欢迎Star和完善,大家面试可以参照考点复习,希望我们一起有点东西。 前前言 为啥今天有个前前言呢? 因为你们的丙丙啊,昨天有牌面了哟,直接被微信官方推荐,知乎推荐,也就仅仅是还行吧(心里乐开花)
Java工作4年来应聘要16K最后没要,细节如下。。。
前奏: 今天2B哥和大家分享一位前几天面试的一位应聘者,工作4年26岁,统招本科。 以下就是他的简历和面试情况。 基本情况: 专业技能: 1、&nbsp;熟悉Sping了解SpringMVC、SpringBoot、Mybatis等框架、了解SpringCloud微服务 2、&nbsp;熟悉常用项目管理工具:SVN、GIT、MAVEN、Jenkins 3、&nbsp;熟悉Nginx、tomca
Python爬虫精简步骤1 获取数据
爬虫的工作分为四步: 1.获取数据。爬虫程序会根据我们提供的网址,向服务器发起请求,然后返回数据。 2.解析数据。爬虫程序会把服务器返回的数据解析成我们能读懂的格式。 3.提取数据。爬虫程序再从中提取出我们需要的数据。 4.储存数据。爬虫程序把这些有用的数据保存起来,便于你日后的使用和分析。 这一篇的内容就是:获取数据。 首先,我们将会利用一个强大的库——requests来获取数据。 在电脑上安装
Python绘图,圣诞树,花,爱心 | Turtle篇
1.画圣诞树 import turtle screen = turtle.Screen() screen.setup(800,600) circle = turtle.Turtle() circle.shape('circle') circle.color('red') circle.speed('fastest') circle.up() square = turtle.Turtle()
作为一个程序员,CPU的这些硬核知识你必须会!
CPU对每个程序员来说,是个既熟悉又陌生的东西? 如果你只知道CPU是中央处理器的话,那可能对你并没有什么用,那么作为程序员的我们,必须要搞懂的就是CPU这家伙是如何运行的,尤其要搞懂它里面的寄存器是怎么一回事,因为这将让你从底层明白程序的运行机制。 随我一起,来好好认识下CPU这货吧 把CPU掰开来看 对于CPU来说,我们首先就要搞明白它是怎么回事,也就是它的内部构造,当然,CPU那么牛的一个东
破14亿,Python分析我国存在哪些人口危机!
2020年1月17日,国家统计局发布了2019年国民经济报告,报告中指出我国人口突破14亿。 猪哥的朋友圈被14亿人口刷屏,但是很多人并没有看到我国复杂的人口问题:老龄化、男女比例失衡、生育率下降、人口红利下降等。 今天我们就来分析一下我们国家的人口数据吧! 更多有趣分析教程,扫描下方二维码关注vx公号「裸睡的猪」 即可查看! 一、背景 1.人口突破14亿 2020年1月17日,国家统计局发布
听说想当黑客的都玩过这个Monyer游戏(1~14攻略)
第零关 进入传送门开始第0关(游戏链接) 请点击链接进入第1关: 连接在左边→ ←连接在右边 看不到啊。。。。(只能看到一堆大佬做完的留名,也能看到菜鸡的我,在后面~~) 直接fn+f12吧 &lt;span&gt;连接在左边→&lt;/span&gt; &lt;a href="first.php"&gt;&lt;/a&gt; &lt;span&gt;←连接在右边&lt;/span&gt; o
在家远程办公效率低?那你一定要收好这个「在家办公」神器!
相信大家都已经收到国务院延长春节假期的消息,接下来,在家远程办公可能将会持续一段时间。 但是问题来了。远程办公不是人在电脑前就当坐班了,相反,对于沟通效率,文件协作,以及信息安全都有着极高的要求。有着非常多的挑战,比如: 1在异地互相不见面的会议上,如何提高沟通效率? 2文件之间的来往反馈如何做到及时性?如何保证信息安全? 3如何规划安排每天工作,以及如何进行成果验收? ......
作为一个程序员,内存和磁盘的这些事情,你不得不知道啊!!!
截止目前,我已经分享了如下几篇文章: 一个程序在计算机中是如何运行的?超级干货!!! 作为一个程序员,CPU的这些硬核知识你必须会! 作为一个程序员,内存的这些硬核知识你必须懂! 这些知识可以说是我们之前都不太重视的基础知识,可能大家在上大学的时候都学习过了,但是嘞,当时由于老师讲解的没那么有趣,又加上这些知识本身就比较枯燥,所以嘞,大家当初几乎等于没学。 再说啦,学习这些,也看不出来有什么用啊!
别低估自己的直觉,也别高估自己的智商
所有群全部吵翻天,朋友圈全部沦陷,公众号疯狂转发。这两周没怎么发原创,只发新闻,可能有人注意到了。我不是懒,是文章写了却没发,因为大家的关注力始终在这次的疫情上面,发了也没人看。当然,我...
这个世界上人真的分三六九等,你信吗?
偶然间,在知乎上看到一个问题 一时间,勾起了我深深的回忆。 以前在厂里打过两次工,做过家教,干过辅导班,做过中介。零下几度的晚上,贴过广告,满脸、满手地长冻疮。   再回首那段岁月,虽然苦,但让我学会了坚持和忍耐。让我明白了,在这个世界上,无论环境多么的恶劣,只要心存希望,星星之火,亦可燎原。   下文是原回答,希望能对你能有所启发。   如果我说,这个世界上人真的分三六九等,
为什么听过很多道理,依然过不好这一生?
记录学习笔记是一个重要的习惯,不希望学习过的东西成为过眼云烟。做总结的同时也是一次复盘思考的过程。 本文是根据阅读得到 App上《万维钢·精英日课》部分文章后所做的一点笔记和思考。学习是一个系统的过程,思维模型的建立需要相对完整的学习和思考过程。以下观点是在碎片化阅读后总结的一点心得总结。
B 站上有哪些很好的学习资源?
哇说起B站,在小九眼里就是宝藏般的存在,放年假宅在家时一天刷6、7个小时不在话下,更别提今年的跨年晚会,我简直是跪着看完的!! 最早大家聚在在B站是为了追番,再后来我在上面刷欧美新歌和漂亮小姐姐的舞蹈视频,最近两年我和周围的朋友们已经把B站当作学习教室了,而且学习成本还免费,真是个励志的好平台ヽ(.◕ฺˇд ˇ◕ฺ;)ノ 下面我们就来盘点一下B站上优质的学习资源: 综合类 Oeasy: 综合
如何优雅地打印一个Java对象?
你好呀,我是沉默王二,一个和黄家驹一样身高,和刘德华一样颜值的程序员。虽然已经写了十多年的 Java 代码,但仍然觉得自己是个菜鸟(请允许我惭愧一下)。 在一个月黑风高的夜晚,我思前想后,觉得再也不能这么蹉跎下去了。于是痛下决心,准备通过输出的方式倒逼输入,以此来修炼自己的内功,从而进阶成为一名真正意义上的大神。与此同时,希望这些文章能够帮助到更多的读者,让大家在学习的路上不再寂寞、空虚和冷。 ...
雷火神山直播超两亿,Web播放器事件监听是怎么实现的?
Web播放器解决了在手机浏览器和PC浏览器上播放音视频数据的问题,让视音频内容可以不依赖用户安装App,就能进行播放以及在社交平台进行传播。在视频业务大数据平台中,播放数据的统计分析非常重要,所以Web播放器在使用过程中,需要对其内部的数据进行收集并上报至服务端,此时,就需要对发生在其内部的一些播放行为进行事件监听。 那么Web播放器事件监听是怎么实现的呢? 01 监听事件明细表 名
3万字总结,Mysql优化之精髓
本文知识点较多,篇幅较长,请耐心学习 MySQL已经成为时下关系型数据库产品的中坚力量,备受互联网大厂的青睐,出门面试想进BAT,想拿高工资,不会点MySQL优化知识,拿offer的成功率会大大下降。 为什么要优化 系统的吞吐量瓶颈往往出现在数据库的访问速度上 随着应用程序的运行,数据库的中的数据会越来越多,处理时间会相应变慢 数据是存放在磁盘上的,读写速度无法和内存相比 如何优化 设计
Linux 命令(122)—— watch 命令
1.命令简介 2.命令格式 3.选项说明 4.常用示例 参考文献 [1] watch(1) manual
Linux 命令(121)—— cal 命令
1.命令简介 2.命令格式 3.选项说明 4.常用示例 参考文献 [1] cal(1) manual
记jsp+servlet+jdbc实现的新闻管理系统
1.工具:eclipse+SQLyog 2.介绍:实现的内容就是显示新闻的基本信息,然后一个增删改查的操作。 3.数据库表设计 列名 中文名称 数据类型 长度 非空 newsId 文章ID int 11 √ newsTitle 文章标题 varchar 20 √ newsContent 文章内容 text newsStatus 是否审核 varchar 10 news...
Python新型冠状病毒疫情数据自动爬取+统计+发送报告+数据屏幕(三)发送篇
今天介绍的项目是使用 Itchat 发送统计报告 项目功能设计: 定时爬取疫情数据存入Mysql 进行数据分析制作疫情报告 使用itchat给亲人朋友发送分析报告(本文) 基于Django做数据屏幕 使用Tableau做数据分析 来看看最终效果 目前已经完成,预计2月12日前更新 使用 itchat 发送数据统计报告 itchat 是一个基于 web微信的一个框架,但微信官方并不允
相关热词 c# 识别回车 c#生成条形码ean13 c#子控制器调用父控制器 c# 写大文件 c# 浏览pdf c#获取桌面图标的句柄 c# list反射 c# 句柄 进程 c# 倒计时 线程 c# 窗体背景色
立即提问