xxl-job与hadoop同时使用 5C
  1. xxl-job如何与hadoop的分布式框架同时用?有人用过吗,有没有一些比较好的资料?期待大神们的回复。

1个回答

建议这种问题直接联系 xxl-job 的作者https://github.com/xuxueli/xxl-job

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
win10下编译hadoop eclipse plugin报错
win10下编译hadoop eclipse plugin报错,请求各位大佬帮忙看一下 ``` D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin>ant jar -Dversion=2.8.3 -Declipse.home=C:\Users\Daybr\eclipse\java-neon\eclipse -Dhadoop.home=D:\hadoop-2.8.3\hadoop-2.8.3 Buildfile: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml check-contrib: init: [echo] contrib: eclipse-plugin init-contrib: ivy-probe-antlib: ivy-init-antlib: ivy-init: [ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ :: [ivy:configure] :: loading settings :: file = D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\ivy\ivysettings.xml ivy-resolve-common: ivy-retrieve-common: [ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead [ivy:cachepath] :: loading settings :: file = D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\ivy\ivysettings.xml compile: [echo] contrib: eclipse-plugin [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 45 source files to D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\build\contrib\eclipse-plugin\classes [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\Activator.java:22: 错误: 程序包org.eclipse.ui.plugin不存在 [javac] import org.eclipse.ui.plugin.AbstractUIPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\Activator.java:28: 错误: 找不到符号 [javac] public class Activator extends AbstractUIPlugin { [javac] ^ [javac] 符号: 类 AbstractUIPlugin [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ErrorMessageDialog.java:22: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Display; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:21: 错误: 程序包org.eclipse.debug.ui不存在 [javac] import org.eclipse.debug.ui.IDebugUIConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:22: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.JavaUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:23: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IFolderLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:24: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IPageLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:25: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IPerspectiveFactory; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:26: 错误: 程序包org.eclipse.ui.console不存在 [javac] import org.eclipse.ui.console.IConsoleConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:34: 错误: 找不到符号 [javac] public class HadoopPerspectiveFactory implements IPerspectiveFactory { [javac] ^ [javac] 符号: 类 IPerspectiveFactory [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\HadoopPerspectiveFactory.java:36: 错误: 找不到符号 [javac] public void createInitialLayout(IPageLayout layout) { [javac] ^ [javac] 符号: 类 IPageLayout [javac] 位置: 类 HadoopPerspectiveFactory [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:25: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.FileLocator; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:26: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:28: 错误: 程序包org.eclipse.swt.graphics不存在 [javac] import org.eclipse.swt.graphics.Image; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:29: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.ISharedImages; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:30: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.PlatformUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:31: 错误: 程序包org.eclipse.ui.plugin不存在 [javac] import org.eclipse.ui.plugin.AbstractUIPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:46: 错误: 找不到符号 [javac] private ISharedImages sharedImages = [javac] ^ [javac] 符号: 类 ISharedImages [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:70: 错误: 找不到符号 [javac] public static Image getImage(String name) { [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:95: 错误: 找不到符号 [javac] private Map<String, Image> imageMap = new HashMap<String, Image>(); [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\ImageLibrary.java:154: 错误: 找不到符号 [javac] private Image getImageByName(String name) { [javac] ^ [javac] 符号: 类 Image [javac] 位置: 类 ImageLibrary [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:29: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:30: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProjectNature; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:31: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:32: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.NullProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:33: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:34: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.QualifiedName; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:35: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IClasspathEntry; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:36: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IJavaProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:37: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.JavaCore; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:44: 错误: 找不到符号 [javac] public class MapReduceNature implements IProjectNature { [javac] ^ [javac] 符号: 类 IProjectNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:48: 错误: 找不到符号 [javac] private IProject project; [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:56: 错误: 找不到符号 [javac] public void configure() throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:130: 错误: 找不到符号 [javac] public void deconfigure() throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:137: 错误: 找不到符号 [javac] public IProject getProject() { [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\MapReduceNature.java:145: 错误: 找不到符号 [javac] public void setProject(IProject project) { [javac] ^ [javac] 符号: 类 IProject [javac] 位置: 类 MapReduceNature [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:21: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IFile; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:22: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:23: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:24: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IJavaElement; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:25: 错误: 程序包org.eclipse.jdt.internal.ui.wizards不存在 [javac] import org.eclipse.jdt.internal.ui.wizards.NewElementWizard; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:28: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.INewWizard; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:29: 错误: 程序包org.eclipse.ui不存在 [javac] import org.eclipse.ui.IWorkbench; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:36: 错误: 找不到符号 [javac] public class NewDriverWizard extends NewElementWizard implements INewWizard, [javac] ^ [javac] 符号: 类 NewElementWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:36: 错误: 找不到符号 [javac] public class NewDriverWizard extends NewElementWizard implements INewWizard, [javac] ^ [javac] 符号: 类 INewWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:23: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:24: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.FileLocator; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:25: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:26: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IStatus; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:27: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:28: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.IType; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:29: 错误: 程序包org.eclipse.jdt.core不存在 [javac] import org.eclipse.jdt.core.JavaModelException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:30: 错误: 程序包org.eclipse.jdt.core.search不存在 [javac] import org.eclipse.jdt.core.search.SearchEngine; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:31: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.IJavaElementSearchConstants; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:32: 错误: 程序包org.eclipse.jdt.ui不存在 [javac] import org.eclipse.jdt.ui.JavaUI; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:33: 错误: 程序包org.eclipse.jdt.ui.wizards不存在 [javac] import org.eclipse.jdt.ui.wizards.NewTypeWizardPage; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:38: 错误: 程序包org.eclipse.swt不存在 [javac] import org.eclipse.swt.SWT; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:39: 错误: 程序包org.eclipse.swt.layout不存在 [javac] import org.eclipse.swt.layout.GridData; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:40: 错误: 程序包org.eclipse.swt.layout不存在 [javac] import org.eclipse.swt.layout.GridLayout; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:41: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Button; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:42: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Composite; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:43: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Event; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:44: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Label; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:45: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Listener; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:46: 错误: 程序包org.eclipse.swt.widgets不存在 [javac] import org.eclipse.swt.widgets.Text; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:47: 错误: 程序包org.eclipse.ui.dialogs不存在 [javac] import org.eclipse.ui.dialogs.SelectionDialog; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:54: 错误: 找不到符号 [javac] public class NewDriverWizardPage extends NewTypeWizardPage { [javac] ^ [javac] 符号: 类 NewTypeWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:43: 错误: 找不到符号 [javac] public void run(IProgressMonitor monitor) { [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:60: 错误: 找不到符号 [javac] public void init(IWorkbench workbench, IStructuredSelection selection) { [javac] ^ [javac] 符号: 类 IWorkbench [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:90: 错误: 找不到符号 [javac] protected void finishPage(IProgressMonitor monitor) [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:91: 错误: 找不到符号 [javac] throws InterruptedException, CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizard.java:96: 错误: 找不到符号 [javac] public IJavaElement getCreatedElement() { [javac] ^ [javac] 符号: 类 IJavaElement [javac] 位置: 类 NewDriverWizard [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:55: 错误: 找不到符号 [javac] private Button isCreateMapMethod; [javac] ^ [javac] 符号: 类 Button [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:57: 错误: 找不到符号 [javac] private Text reducerText; [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:59: 错误: 找不到符号 [javac] private Text mapperText; [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:85: 错误: 找不到符号 [javac] public void createType(IProgressMonitor monitor) throws CoreException, [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:85: 错误: 找不到符号 [javac] public void createType(IProgressMonitor monitor) throws CoreException, [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:91: 错误: 找不到符号 [javac] protected void createTypeMembers(final IType newType, ImportsManager imports, [javac] ^ [javac] 符号: 类 IType [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:91: 错误: 找不到符号 [javac] protected void createTypeMembers(final IType newType, ImportsManager imports, [javac] ^ [javac] 符号: 类 ImportsManager [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:92: 错误: 找不到符号 [javac] final IProgressMonitor monitor) throws CoreException { [javac] ^ [javac] 符号: 类 IProgressMonitor [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:92: 错误: 找不到符号 [javac] final IProgressMonitor monitor) throws CoreException { [javac] ^ [javac] 符号: 类 CoreException [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:145: 错误: 找不到符号 [javac] public void createControl(Composite parent) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:199: 错误: 找不到符号 [javac] private void createMapperControls(Composite composite) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:204: 错误: 找不到符号 [javac] private void createReducerControls(Composite composite) { [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:209: 错误: 找不到符号 [javac] private Text createBrowseClassControl(final Composite composite, [javac] ^ [javac] 符号: 类 Composite [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewDriverWizardPage.java:209: 错误: 找不到符号 [javac] private Text createBrowseClassControl(final Composite composite, [javac] ^ [javac] 符号: 类 Text [javac] 位置: 类 NewDriverWizardPage [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:29: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProject; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:30: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.IProjectDescription; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:31: 错误: 程序包org.eclipse.core.resources不存在 [javac] import org.eclipse.core.resources.ResourcesPlugin; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:32: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.CoreException; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:33: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IConfigurationElement; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:34: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.IExecutableExtension; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:36: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.NullProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:37: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.Path; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:38: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.QualifiedName; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:39: 错误: 程序包org.eclipse.core.runtime不存在 [javac] import org.eclipse.core.runtime.SubProgressMonitor; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:40: 错误: 程序包org.eclipse.jdt.ui.wizards不存在 [javac] import org.eclipse.jdt.ui.wizards.NewJavaProjectWizardPage; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:49: 错误: 程序包org.eclipse.swt不存在 [javac] import org.eclipse.swt.SWT; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:50: 错误: 程序包org.eclipse.swt.events不存在 [javac] import org.eclipse.swt.events.SelectionEvent; [javac] ^ [javac] D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\NewMapReduceProjectWizard.java:51: 错误: 程序包org.eclipse.swt.events不存在 [javac] import org.eclipse.swt.events.SelectionListener; [javac] ^ [javac] 注: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\dfs\DFSFolder.java使用或覆盖了已过时的 API。 [javac] 注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。 [javac] 注: D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\src\java\org\apache\hadoop\eclipse\actions\DFSActionImpl.java使用了未经检查或不安全的操作。 [javac] 注: 有关详细信息, 请使用 -Xlint:unchecked 重新编译。 [javac] 注: 某些消息已经过简化; 请使用 -Xdiags:verbose 重新编译以获得完整输出 [javac] 100 个错误 BUILD FAILED D:\hadoop2x-eclipse-plugin-master-master\hadoop2x-eclipse-plugin-master-master\src\contrib\eclipse-plugin\build.xml:76: Compile failed; see the compiler error output for details. Total time: 4 seconds ```
Ccentos6 编译hadoop2.6 b报错
http://my.csdn.net/my/mycsdnFailed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program "cmake" (in directory "/usr/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/target/native"): error=2, No such file or directory [ERROR] around Ant part ...<exec dir="/usr/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:125 in /usr/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
osx下编译hadoop-2.5.2-src出错
[exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:9: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration] [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) { [exec] ^ [exec] 1 warning and 2 errors generated. [exec] make[2]: *** [CMakeFiles/test_libhdfs_vecsum.dir/main/native/libhdfs/test/vecsum.c.o] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_vecsum.dir/all] Error 2 [exec] make: *** [all] Error 2
osx下编译hadoop2.5.2的时候报错Ant BuildException
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (site) on project hadoop-common: An Ant BuildException has occured: stylesheet /Users/narayan/Downloads/hadoop-2.5.2-src/hadoop-common-project/hadoop-common/${env.FINDBUGS_HOME}/src/xsl/default.xsl doesn't exist. [ERROR] around Ant part ...<xslt style="${env.FINDBUGS_HOME}/src/xsl/default.xsl" in="/Users/narayan/Downloads/hadoop-2.5.2-src/hadoop-common-project/hadoop-common/target/findbugsXml.xml" out="/Users/narayan/Downloads/hadoop-2.5.2-src/hadoop-common-project/hadoop-common/target/site/findbugs.html"/>... @ 44:275 in /Users/narayan/Downloads/hadoop-2.5.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hadoop-common 网上查是说少了findbugs?但是我brew安装了findbugs之后依然报错。然后说是在安装目录下$ ant 然后 Buildfile: build.xml does not exist! Build failed 还是需要其他的什么设置么?
hadoop单词统计报错Job job_1581768459583_0001 failed
3个节点hadoop01、hadoop02、hadoop03 hadoop01是主节点 hadoop01、hadoop02、hadoop03是从节点,目前集群已搭建好,jps查看三个节点运行都很正常,而且UI也能正常显示,但是使用hadoop自带的hadoop-mapreduce-examples-2.7.4.jar的wordcount进行单词统计时报错如下,请高人指点,看不懂呀: ```[root@hadoop01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.4.jar wordcount /wordcount/input /wordcount/output 20/02/15 20:14:25 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.233.132:8032 20/02/15 20:14:27 INFO input.FileInputFormat: Total input paths to process : 1 20/02/15 20:14:27 INFO mapreduce.JobSubmitter: number of splits:1 20/02/15 20:14:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1581768459583_0001 20/02/15 20:14:28 INFO impl.YarnClientImpl: Submitted application application_1581768459583_0001 20/02/15 20:14:28 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1581768459583_0001/ 20/02/15 20:14:28 INFO mapreduce.Job: Running job: job_1581768459583_0001 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 running in uber mode : false 20/02/15 20:15:38 INFO mapreduce.Job: map 0% reduce 0% 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 failed with state FAILED due to: Application application_1581768459583_0001 failed 2 times due to Error launching appattempt_1581768459583_0001_000002. Got exception: java.io.IOException: Failed on local exception: java.io.IOException: java.io.IOException: Connection reset by peer; Host Details : local host is: "hadoop01.com/79.124.78.101"; destination host is: "79.124.78.101":43276; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy83.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy84.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.io.IOException: Connection reset by peer at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:651) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 16 more Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726) ... 19 more . Failing the application. 20/02/15 20:15:38 INFO mapreduce.Job: Counters: 0 ```
执行pyspark时出现TypeError如何解决?
打算在WIndows 10环境配置一下Spark,跟着网上的教程一步一步走到最后,执行spark-shell也成功进入Scala环境 ![图片说明](https://img-ask.csdn.net/upload/202002/03/1580724413_275891.png) 但是执行pyspark却提示TypeError: an integer is required (got type bytes) 完整信息: ``` Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 23:11:46) [MSC v.1916 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. Traceback (most recent call last): File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\shell.py", line 31, in <module> from pyspark import SparkConf File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\__init__.py", line 51, in <module> from pyspark.context import SparkContext File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\context.py", line 31, in <module> from pyspark import accumulators File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\accumulators.py", line 97, in <module> from pyspark.serializers import read_int, PickleSerializer File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\serializers.py", line 71, in <module> from pyspark import cloudpickle File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 145, in <module> _cell_set_template_code = _make_cell_set_template_code() File "D:\spark-2.4.4-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code return types.CodeType( TypeError: an integer is required (got type bytes) ``` 请问如何解决这个问题?是不是Python版本太高了?
spring-hadoop.xsd在哪
江湖救急,spring-hadoop.xsd在哪spring-hadoop.xsd在哪spring-hadoop.xsd在哪spring-hadoop.xsd在哪spring-hadoop.xsd在哪spring-hadoop.xsd在哪spring-hadoop.xsd在哪
诡异的Hadoop启动脚本
**跟踪start-dfs.sh脚本启动时,发现极其诡异的一点,如下:** --- Hadoop版本: Apache Hadoop2.6.5 --- **1、start-dfs.sh** start-dfs.sh中有这样语句: ``` "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$NAMENODES" \ --script "$bin/hdfs" start namenode $nameStartOpt ``` 对应执行命令为 ``` hadoop-daemons.sh --config "$HADOOP_CONF_DIR" --hostnames "$NAMENODES" --script "$bin/hdfs" start namenode $nameStartOpt ``` 跟踪到hadoop-daemons.sh --- **2、hadoop-daemons.sh** ``` exec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_PREFIX" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@" ``` 接着执行到这里,最后执行 ``` "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@" ``` 这里的"$@" 为我理解为从[start-dfs.sh]中传过来的 ``` --config "$HADOOP_CONF_DIR" --hostnames "$NAMENODES" --script "$bin/hdfs" start namenode $nameStartOpt ``` 这么一大串参数。 接着跟踪,执行到hadoop-daemon.sh脚本 --- **3、hadoop-daemon.sh** ``` hadoopScript="$HADOOP_PREFIX"/bin/hadoop if [ "--script" = "$1" ] then shift hadoopScript=$1 shift fi startStop=$1 shift command=$1 shift ``` 前后都没有进行过 shift 操作。 这里比较 ``` $1="--script" ``` 是啥意思? 从hadoop-daemons.sh传入到此脚本的参数难道不是应该为 ``` --config "$HADOOP_CONF_DIR" --config "$HADOOP_CONF_DIR" --hostnames "$NAMENODES" --script "$bin/hdfs" start namenode $nameStartOpt ``` 这几个参数吗? 为什么上面的处理过程好像传入的只有 ``` --script "$bin/hdfs" start namenode $nameStartOpt ``` 这个参数? 实在是搞不懂。
求教,hadoop-2.2.0升级hadoop-2.6.0。
最近需要升级hadoop,从hadoop-2.2.0升级到hadoop-2.6.0,根据http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade 提示的来,第一步:./bin/hdfs dfsadmin -rollingUPgrade prepare 就出现了:PREPARE rolling upgrade ... rollingUpgrade: Unknown method rollingUpgrade called on org.apache.hadoop.hdfs.protocol.ClientProtocol protocol.
ubuntu 64位编译hadoop-2.6.0失败,网上看了好久,都没解决,大神速来解救
[exec] CMake Error at /usr/local/share/cmake-2.6/Modules/FindPackageHandleStandardArgs.cmake:52 (MESSAGE): [exec] Could NOT find ZLIB [exec] Call Stack (most recent call first): [exec] /usr/local/share/cmake-2.6/Modules/FindZLIB.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) [exec] CMakeLists.txt:107 (find_package) [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 [ERROR] around Ant part ...<exec dir="/home/cj/workspace/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:139 in /home/cj/workspace/hadoop-2.6.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hadoop-common 好像是 Could NOT find ZLIB引起的
hadoop2.7 本地库问题
我的电脑是ubuntu14.10, 64位的.安装hadoop2.7之后总是会有这个问题 > WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 看网上说,hadoop包含的是本地库是32位的,但是我到lib/native目录里用file命令查看, libhadoop.so.1.0 是64位的啊.下边是file libhadoop.so.1.0.0的输出信息 > libhadoop.so.1.0.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ed024ac48c0f542fa36ddc918a75c51e1c647424, not stripped 看官网上也是说随着发行的是32位的本地库..为什么我查看的结果是64的? 另外还有个问题,编译hadoop的时候,编译一会,总是卡在 > main: [mkdir] Created dir: /home/cirfe/download/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/downloads [get] Getting: http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz [get] To: /home/cirfe/download/hadoop-2.6.0-src/hadoop-common-project/hadoop-kms/downloads/apache-tomcat-6.0.41.tar.gz 就编译不下去了,一直卡在这里. 请教大神这几个问题怎么解决?
编译安装hadoop-2.5.0-rc1,程序包com.sun.javadoc不存在
编译安装hadoop-2.5.0-rc1,报如下错,求指点: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hadoop-annotations: Compilation failure: Compilation failure: [ERROR] /usr/local/src/release-2.5.0-rc1/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/tools/ExcludePrivateAnnotationsJDiffDoclet.java:[20,22] 错误: 程序包com.sun.javadoc不存在
hadoop中wordcount使用IKsegmenter分词器后本地编译运行通过,打成jar包后报IKsegmenter类无法找到
使用eclipse在本地上编译运行的时候没有问题,output能够获得正常结果。 但是使用hadoop运行时就会报错,之前没有使用过ecipse,在网上找了半天,export里面的选项来回试了一下也没成功..以下是报错信息: ``` 19/11/19 20:57:39 INFO client.RMProxy: Connecting to ResourceManager at bigdata-senior01.chybinmy.com/192.168.100.10:8032 19/11/19 20:57:40 INFO input.FileInputFormat: Total input files to process : 1 19/11/19 20:57:40 INFO mapreduce.JobSubmitter: number of splits:1 19/11/19 20:57:41 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/11/19 20:57:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1574165004078_0005 19/11/19 20:57:41 INFO conf.Configuration: resource-types.xml not found 19/11/19 20:57:41 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 19/11/19 20:57:41 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE 19/11/19 20:57:41 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE 19/11/19 20:57:42 INFO impl.YarnClientImpl: Submitted application application_1574165004078_0005 19/11/19 20:57:42 INFO mapreduce.Job: The url to track the job: http://bigdata-senior01.chybinmy.com:8088/proxy/application_1574165004078_0005/ 19/11/19 20:57:42 INFO mapreduce.Job: Running job: job_1574165004078_0005 19/11/19 20:57:49 INFO mapreduce.Job: Job job_1574165004078_0005 running in uber mode : false 19/11/19 20:57:49 INFO mapreduce.Job: map 0% reduce 0% 19/11/19 20:57:53 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171) 19/11/19 20:57:58 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_1, Status : FAILED Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171) 19/11/19 20:58:03 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_2, Status : FAILED Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36) at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171) 19/11/19 20:58:11 INFO mapreduce.Job: map 100% reduce 100% 19/11/19 20:58:12 INFO mapreduce.Job: Job job_1574165004078_0005 failed with state FAILED due to: Task failed task_1574165004078_0005_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0 19/11/19 20:58:12 INFO mapreduce.Job: Counters: 13 Job Counters Failed map tasks=4 Killed reduce tasks=1 Launched map tasks=4 Other local map tasks=3 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=12897 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=12897 Total vcore-milliseconds taken by all map tasks=12897 Total megabyte-milliseconds taken by all map tasks=13206528 Map-Reduce Framework CPU time spent (ms)=0 Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0 [hadoop@bigdata-senior01 hadoop-2.10.0]$ bin/hdfs dfs -ls / Found 4 items drwxr-xr-x - hadoop supergroup 0 2019-11-19 20:13 /input drwxr-xr-x - hadoop supergroup 0 2019-11-19 20:54 /output drwxr-xr-x - hadoop supergroup 0 2019-11-19 20:58 /output2 drwx------ - hadoop supergroup 0 2019-11-19 20:16 /tmp ```
hadoop使用kerberos认证后,hadoop fs -ls命令行无法使用,求大神帮忙
hadoop版本apache hadoop 2.7.3,jdk-1.7 输入hadoop fs -ls,错误信息如下: hadoop@hadoop01 native]$ hadoop fs -ls 17/08/01 01:33:36 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop01/192.168.148.129"; destination host is: "hadoop01":9000; klist查看凭据缓存,是存在的: [hadoop@hadoop01 native]$ klist Ticket cache: KEYRING:persistent:1001:1001 Default principal: hadoop/hadoop01@HADOOP.COM Valid starting Expires Service principal 08/01/2017 01:12:54 08/02/2017 01:12:54 krbtgt/HADOOP.COM@HADOOP.COM 通过http://192.168.148.129:50070/dfshealth.html#tab-overview访问界面也是OK的: Configured Capacity: 55.38 GB DFS Used: 16 KB (0%) Non DFS Used: 11.4 GB DFS Remaining: 43.99 GB (79.42%) Block Pool Used: 16 KB (0%) DataNodes usages% (Min/Median/Max/stdDev): 0.00% / 0.00% / 0.00% / 0.00% Live Nodes 2 (Decommissioned: 0) Dead Nodes 0 (Decommissioned: 0) Decommissioning Nodes 0 Total Datanode Volume Failures 0 (0 B) Number of Under-Replicated Blocks 0 Number of Blocks Pending Deletion 0 Block Deletion Start Time 2017/8/1 上午10:12:21
HDFS启动警告 WARN util.NativeCodeLoader
启动HDFS报入校警告: [hadoop@hadoop tmp]$ start-dfs.sh 15/02/02 20:39:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [hadoop] hadoop: starting namenode, logging to /usr/hadoop-2.5.2/logs/hadoop-hadoop-namenode-hadoop.out localhost: starting datanode, logging to /usr/hadoop-2.5.2/logs/hadoop-hadoop-datanode-hadoop.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/hadoop-2.5.2/logs/hadoop-hadoop-secondarynamenode-hadoop.out 15/02/02 20:40:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 网上查说是liunx版本问题 [hadoop@hadoop ~]$ uname -a Linux hadoop 2.6.32-431.el6.i686 #1 SMP Fri Nov 22 00:26:36 UTC 2013 i686 i686 i386 GNU/Linux JDK版本 [hadoop@hadoop ~]$ java -version java version "1.6.0_24" Java(TM) SE Runtime Environment (build 1.6.0_24-b07) Java HotSpot(TM) Client VM (build 19.1-b02, mixed mode, sharing) Hadoop版本: [hadoop@hadoop ~]$ hadoop version Hadoop 2.5.2 Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r cc72e9b000545b86b75a61f4835eb86d57bfafc0 Compiled by jenkins on 2014-11-14T23:45Z Compiled with protoc 2.5.0 From source with checksum df7537a4faa4658983d397abf4514320 This command was run using /usr/hadoop-2.5.2/share/hadoop/common/hadoop-common-2.5.2.jar 请高手帮忙看下
初入Hadoop,start-all.sh问题
Hadoop 版本:hadoop-2.6.5 环境: ![图片说明](https://img-ask.csdn.net/upload/201711/08/1510104898_307684.png) HODOOP_HOME 目录也不同:都在各自用户的目录下 e.g: /home/yann/hadoop /home/ubuntu01/hadoop /home/ubuntu02/hadoop - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 当我在启动的时候start-all.sh 提示我如下错误: yann@yann-laptop:~/develop/tool/hadoop-2.6.5/sbin$ ./start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [yann-laptop] yann-laptop: namenode running as process 7041. Stop it first. yann@ubuntu01-virtual-machine's password: The authenticity of host 'ubuntu02-virtual-machine (192.168.2.182)' can't be established. ECDSA key fingerprint is 57:ee:92:a8:85:85:ef:16:26:a3:b7:1d:54:77:19:18. Are you sure you want to continue connecting (yes/no)? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - hadoop/etc/hadoop/slaves: ubuntu01-virtual-machine ubuntu02-virtual-machine - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - SSH公钥已经配置好: ssh-rsa XXXXX......XXXXXX yann@yann-laptop ssh-rsa XXXXX......XXXXXX ubuntu01@ubuntu01-virtual-machine ssh-rsa XXXXX......XXXXXX ubuntu02@ubuntu02-virtual-machine - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 从提示来看它使用了yann去连接ubuntu01-virtual-machine。 请问这种情况如何配置(主从机器不同的用户名)?
hadoop简单问题,如何在spring boot中,同时使用这4个jar,能够正常运行
dependencies { compile('org.springframework.boot:spring-boot-starter') compile("org.apache.hadoop:hadoop-common:2.7.3") compile("org.apache.hadoop:hadoop-mapreduce-client-core:2.7.3") compile("org.apache.hadoop:hadoop-hdfs:2.7.3") } 或者这样也行 dependencies { compile('org.springframework.boot:spring-boot-starter') compile 'org.springframework.data:spring-data-hadoop:2.5.0.RELEASE' } 我就是想在spring boot中连接hadoop的hdfs而已,为啥要这样折磨我,请各位大神可以的话新建一个基本的spring boot项目,然后添加一下以上的关于hadoop的包,然后试着运行一下,我已经哭了
hadoop集群,hdfs dfs -ls / 目录出错
搭建了一个hadoop集群,用hdfs dfs -ls /命令,列出的是本地系统的根目录。 用hdfs dfs -ls hdfs://servicename/ 列出的目录才是hdfs上的目录,可能是什么原因? 执行hive创建的目录也是在本地系统目录上。 集群的配置如下 集群规划: 主机名 IP 安装的软件 运行的进程 hadoop01 192.168.175.129 jdk、hadoop NameNode、DFSZKFailoverController(zkfc) hadoop02 192.168.175.127 jdk、hadoop NameNode、DFSZKFailoverController(zkfc) hadoop03 192.168.175.126 jdk、hadoop ResourceManager hadoop04 192.168.175.125 jdk、hadoop ResourceManager hadoop05 192.168.175.124 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain hadoop06 192.168.175.123 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain hadoop07 192.168.175.122 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain windows:NLB LINUX:LVS 1.liunx虚拟机安装后,虚拟机连接模式要选择host-only模式。然后分配IP(以hadoop01为例) DEVICE="eth0" BOOTPROTO="static" ### HWADDR="00:0C:29:3C:BF:E7" IPV6INIT="yes" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c" IPADDR="192.168.175.129" ### NETMASK="255.255.255.0" ### GATEWAY="192.168.175.1" ### 2.修改主机名: vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop01 ### 3.关闭防火墙: #查看防火墙状态 service iptables status #关闭防火墙 service iptables stop #查看防火墙开机启动状态 chkconfig iptables --list #关闭防火墙开机启动 chkconfig iptables off 4.免登录配置: #生成ssh免登陆密钥 #进入到我的home目录 cd ~/.ssh ssh-keygen -t rsa (四个回车) 执行完这个命令后,会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥) 将公钥拷贝到要免登陆的机器上 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 或 若报错ssh-copy-id: ERROR: No identities found,是因为找不到公钥路径,加上-i然后再加上路径即可 则用 $ ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote_ip 5.主机IP映射关系(/etc/hosts每台机器上都要配置全部映射关系) 192.168.175.129 hadoop01 192.168.175.127 hadoop02 192.168.175.126 hadoop03 192.168.175.125 hadoop04 192.168.175.124 hadoop05 192.168.175.123 hadoop06 192.168.175.122 hadoop07 6./etc/profile下配置java环境变量: export JAVA_HOME=/lichangwu/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin #刷新profile source /etc/profile 若版本报错,vi /etc/selinux/config,设置SELINUX=disabled,然后重启虚拟机 7.安装zookeeper: 1.安装配置zooekeeper集群(在hadoop05上): 1.1解压 tar -zxvf zookeeper-3.4.6.tar.gz -C /lichangwu/ 1.2修改配置 cd /lichangwu/zookeeper-3.4.6/conf/ cp zoo_sample.cfg zoo.cfg vim zoo.cfg 修改:dataDir=/lichangwu/zookeeper-3.4.6/tmp 在最后添加: server.1=hadoop05:2888:3888 server.2=hadoop06:2888:3888 server.3=hadoop07:2888:3888 保存退出 然后创建一个tmp文件夹 mkdir /lichangwu/zookeeper-3.4.6/tmp 再创建一个空文件 touch /lichangwu/zookeeper-3.4.6/tmp/myid 最后向该文件写入ID echo 1 > /lichangwu/zookeeper-3.4.6/tmp/myid 1.3将配置好的zookeeper拷贝到其他节点(首先分别在hadoop06、hadoop07根目录下创建一个lichangwu目录:mkdir /lichangwu) scp -r /lichangwu/zookeeper-3.4.6/ hadoop06:/lichangwu/ scp -r /lichangwu/zookeeper-3.4.6/ hadoop07:/lichangwu/ 注意:修改hadoop06、hadoop07对应/lichangwu/zookeeper-3.4.6/tmp/myid内容 itcast06: echo 2 > /lichangwu/zookeeper-3.4.6/tmp/myid itcast07: echo 3 > /lichangwu/zookeeper-3.4.6/tmp/myid 8.安装配置hadoop集群(在hadoop01上操作): 2.1解压 tar -zxvf hadoop-2.4.1.tar.gz -C /lichangwu/ 2.2配置HDFS(hadoop2.0所有的配置文件都在$HADOOP_HOME/etc/hadoop目录下) #将hadoop添加到环境变量中 vim /etc/profile export JAVA_HOME=/lichangwu/jdk1.7.0_79 export HADOOP_HOME=/lichangwu/hadoop-2.4.1 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin #hadoop2.0的配置文件全部在$HADOOP_HOME/etc/hadoop下 cd /lichangwu/hadoop-2.4.1/etc/hadoop 2.2.1修改hadoo-env.sh export JAVA_HOME=/lichangwu/jdk1.7.0_79 2.2.2修改core-site.xml <configuration> <!-- 指定hdfs的nameservice为ns1 --> <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property> <!-- 指定hadoop临时目录 --> <property> <name>hadoop.tmp.dir</name> <value>/lichangwu/hadoop-2.4.1/tmp</value> </property> <!-- 指定zookeeper地址 --> <property> <name>ha.zookeeper.quorum</name> <value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value> </property> </configuration> 2.2.3修改hdfs-site.xml <configuration> <!--指定hdfs的nameservice为ns1,需要和core-site.xml中的保持一致 --> <property> <name>dfs.nameservices</name> <value>ns1</value> </property> <!-- ns1下面有两个NameNode,分别是nn1,nn2 --> <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>hadoop01:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>hadoop01:50070</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>hadoop02:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>hadoop02:50070</value> </property> <!-- 指定NameNode的元数据在JournalNode上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop05:8485;hadoop06:8485;hadoop07:8485/ns1</value> </property> <!-- 指定JournalNode在本地磁盘存放数据的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/lichangwu/hadoop-2.4.1/journal</value> </property> <!-- 开启NameNode失败自动切换 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失败自动切换实现方式 --> <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔离机制方法,多个机制用换行分割,即每个机制暂用一行--> <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property> <!-- 使用sshfence隔离机制时需要ssh免登陆 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property> <!-- 配置sshfence隔离机制超时时间 --> <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> </configuration> 2.2.4修改mapred-site.xml <configuration> <!-- 指定mr框架为yarn方式 --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> 2.2.5修改yarn-site.xml <configuration> <!-- 开启RM高可靠 --> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <!-- 指定RM的cluster id --> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property> <!-- 指定RM的名字 --> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <!-- 分别指定RM的地址 --> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>hadoop03</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>hadoop04</value> </property> <!-- 指定zk集群地址 --> <property> <name>yarn.resourcemanager.zk-address</name> <value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> 2.2.6修改slaves(slaves是指定子节点的位置,因为要在itcast01上启动HDFS、在itcast03启动yarn, 所以itcast01上的slaves文件指定的是datanode的位置,itcast03上的slaves文件指定的是nodemanager的位置) hadoop05 hadoop06 hadoop07 2.2.7配置免密码登陆 #首先要配置itcast01到hadoop02、hadoop03、hadoop04、hadoop05、hadoop06、hadoop07的免密码登陆 #在hadoop01上生产一对钥匙 ssh-keygen -t rsa #将公钥拷贝到其他节点,包括自己 ssh-coyp-id hadoop01 ssh-coyp-id hadoop02 ssh-coyp-id hadoop03 ssh-coyp-id hadoop04 ssh-coyp-id hadoop05 ssh-coyp-id hadoop06 ssh-coyp-id hadoop07 #配置hadoop03到hadoop04、hadoop05、hadoop06、hadoop07的免密码登陆 #在hadoop03上生产一对钥匙 ssh-keygen -t rsa #将公钥拷贝到其他节点 ssh-coyp-id hadoop04 ssh-coyp-id hadoop05 ssh-coyp-id hadoop06 ssh-coyp-id hadoop07 #注意:两个namenode之间要配置ssh免密码登陆,别忘了配置hadoop02到hadoop01的免登陆 在hadoop02上生产一对钥匙 ssh-keygen -t rsa ssh-coyp-id -i hadoop01 2.4将配置好的hadoop拷贝到其他节点 scp -r hadoop-2.4.1/ hadoop02:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop03:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop04:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop05:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop06:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop07:/lichangwu/hadoop-2.4.1/ ###注意:严格按照下面的步骤 2.5启动zookeeper集群(分别在hadoop05、hadoop06、hadoop07上启动zk) cd /lichangwu/zookeeper-3.4.6/bin/ ./zkServer.sh start #查看状态:一个leader,两个follower ./zkServer.sh status 2.6启动journalnode(分别在在hadoop05、hadoop06、hadoop07上执行) cd /lichangwu/hadoop-2.4.1 sbin/hadoop-daemon.sh start journalnode #运行jps命令检验,hadoop05、hadoop06、hadoop07上多了JournalNode进程 2.7格式化HDFS #在hadoop01上执行命令: hdfs namenode -format #格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件, 这里我配置的是/lichangwu/hadoop-2.4.1/tmp,然后将/lichangwu/hadoop-2.4.1/tmp拷贝到hadoop02的/lichangwu/hadoop-2.4.1/下。 scp -r tmp/ hadoop02:/lichangwu/hadoop-2.4.1/ 2.8格式化ZK(在hadoop01上执行即可) hdfs zkfc -formatZK 2.9启动HDFS(在hadoop01上执行) sbin/start-dfs.sh 2.10启动YARN(#####注意#####:是在hadoop03上执行start-yarn.sh, 如果hadoop04上没有启动成功,则在hadoop04上再启动一次start-yarn.sh; 把namenode和resourcemanager分开是因为性能问题,因为他们都要占用大量资源,所以把他们分开了,他们分开了就要分别在不同的机器上启动) sbin/start-yarn.sh 到此,hadoop-2.4.1配置完毕,可以统计浏览器访问: http://192.168.175.129:50070 NameNode 'hadoop01:9000' (active) http://192.168.175.127:50070 NameNode 'hadoop02:9000' (standby)
centos6.8搭建hadoop2.X伪分布式无法启动namenode
能够格式化节点信息,但是namenode无法启动。在日志中出现如下错误 ``` STARTUP_MSG: build = Unknown -r Unknown; compiled by 'root' on 2017-05-22T10:49Z STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 2020-01-31 16:37:06,931 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 2020-01-31 16:37:06,935 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode [] 2020-01-31 16:37:07,161 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2020-01-31 16:37:07,233 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2020-01-31 16:37:07,233 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system started 2020-01-31 16:37:07,236 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is hdfs://hadoop101:9000 2020-01-31 16:37:07,236 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use hadoop101:9000 to access this namenode/service. 2020-01-31 16:37:07,409 INFO org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for hdfs at: http://huawei_mate_10-53013e4c60:50070 2020-01-31 16:37:07,457 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2020-01-31 16:37:07,464 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2020-01-31 16:37:07,469 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.namenode is not defined 2020-01-31 16:37:07,473 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2020-01-31 16:37:07,475 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode. java.lang.IllegalArgumentException: The value of property bind.address must not be null at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1134) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1115) at org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:398) at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:351) at org.apache.hadoop.http.HttpServer2.<init>(HttpServer2.java:114) at org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:290) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:126) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:638) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554) 2020-01-31 16:37:07,477 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2020-01-31 16:37:07,479 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop101/192.168.117.101 ************************************************************/ ``` 主要的报错信息是 java.lang.IllegalArgumentException: The value of property bind.address must not be null core-site.xml的配置信息 <configuration> <!-- 指定HDFS中NameNode的地址 --> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop101:9000</value> </property> <!-- hadoop101已经在hosts文件中配置 --> <!-- 指定Hadoop运行时产生文件的存储目录 --> <property> <name>hadoop.tmp.dir</name> <value>/opt/module/hadoop-2.7.2/data/tmp</value> </property> </configuration> 希望大神能够帮忙解答一下。万分感谢感谢
终于明白阿里百度这样的大公司,为什么面试经常拿ThreadLocal考验求职者了
点击上面↑「爱开发」关注我们每晚10点,捕获技术思考和创业资源洞察什么是ThreadLocalThreadLocal是一个本地线程副本变量工具类,各个线程都拥有一份线程私...
《奇巧淫技》系列-python!!每天早上八点自动发送天气预报邮件到QQ邮箱
将代码部署服务器,每日早上定时获取到天气数据,并发送到邮箱。 也可以说是一个小人工智障。 思路可以运用在不同地方,主要介绍的是思路。
面试官问我:什么是消息队列?什么场景需要他?用了会出现什么问题?
你知道的越多,你不知道的越多 点赞再看,养成习惯 GitHub上已经开源 https://github.com/JavaFamily 有一线大厂面试点脑图、个人联系方式和人才交流群,欢迎Star和完善 前言 消息队列在互联网技术存储方面使用如此广泛,几乎所有的后端技术面试官都要在消息队列的使用和原理方面对小伙伴们进行360°的刁难。 作为一个在互联网公司面一次拿一次Offer的面霸...
8年经验面试官详解 Java 面试秘诀
作者 |胡书敏 责编 | 刘静 出品 | CSDN(ID:CSDNnews) 本人目前在一家知名外企担任架构师,而且最近八年来,在多家外企和互联网公司担任Java技术面试官,前后累计面试了有两三百位候选人。在本文里,就将结合本人的面试经验,针对Java初学者、Java初级开发和Java开发,给出若干准备简历和准备面试的建议。 Java程序员准备和投递简历的实...
究竟你适不适合买Mac?
我清晰的记得,刚买的macbook pro回到家,开机后第一件事情,就是上了淘宝网,花了500元钱,找了一个上门维修电脑的师傅,上门给我装了一个windows系统。。。。。。 表砍我。。。 当时买mac的初衷,只是想要个固态硬盘的笔记本,用来运行一些复杂的扑克软件。而看了当时所有的SSD笔记本后,最终决定,还是买个好(xiong)看(da)的。 已经有好几个朋友问我mba怎么样了,所以今天尽量客观...
MyBatis研习录(01)——MyBatis概述与入门
MyBatis 是一款优秀的持久层框架,它支持定制化 SQL、存储过程以及高级映射。MyBatis原本是apache的一个开源项目iBatis, 2010年该项目由apache software foundation 迁移到了google code并改名为MyBatis 。2013年11月MyBatis又迁移到Github。
程序员一般通过什么途径接私活?
二哥,你好,我想知道一般程序猿都如何接私活,我也想接,能告诉我一些方法吗? 上面是一个读者“烦不烦”问我的一个问题。其实不止是“烦不烦”,还有很多读者问过我类似这样的问题。 我接的私活不算多,挣到的钱也没有多少,加起来不到 20W。说实话,这个数目说出来我是有点心虚的,毕竟太少了,大家轻喷。但我想,恰好配得上“一般程序员”这个称号啊。毕竟苍蝇再小也是肉,我也算是有经验的人了。 唾弃接私活、做外...
Python爬虫爬取淘宝,京东商品信息
小编是一个理科生,不善长说一些废话。简单介绍下原理然后直接上代码。 使用的工具(Python+pycharm2019.3+selenium+xpath+chromedriver)其中要使用pycharm也可以私聊我selenium是一个框架可以通过pip下载 pip installselenium -ihttps://pypi.tuna.tsinghua.edu.cn/simple/ ...
阿里程序员写了一个新手都写不出的低级bug,被骂惨了。
这种新手都不会范的错,居然被一个工作好几年的小伙子写出来,差点被当场开除了。
Java工作4年来应聘要16K最后没要,细节如下。。。
前奏: 今天2B哥和大家分享一位前几天面试的一位应聘者,工作4年26岁,统招本科。 以下就是他的简历和面试情况。 基本情况: 专业技能: 1、&nbsp;熟悉Sping了解SpringMVC、SpringBoot、Mybatis等框架、了解SpringCloud微服务 2、&nbsp;熟悉常用项目管理工具:SVN、GIT、MAVEN、Jenkins 3、&nbsp;熟悉Nginx、tomca...
Python爬虫精简步骤1 获取数据
爬虫,从本质上来说,就是利用程序在网上拿到对我们有价值的数据。 爬虫能做很多事,能做商业分析,也能做生活助手,比如:分析北京近两年二手房成交均价是多少?广州的Python工程师平均薪资是多少?北京哪家餐厅粤菜最好吃?等等。 这是个人利用爬虫所做到的事情,而公司,同样可以利用爬虫来实现巨大的商业价值。比如你所熟悉的搜索引擎——百度和谷歌,它们的核心技术之一也是爬虫,而且是超级爬虫。 从搜索巨头到人工...
Python绘图,圣诞树,花,爱心 | Turtle篇
每周每日,分享Python实战代码,入门资料,进阶资料,基础语法,爬虫,数据分析,web网站,机器学习,深度学习等等。 公众号回复【进群】沟通交流吧,QQ扫码进群学习吧 微信群 QQ群 1.画圣诞树 import turtle screen = turtle.Screen() screen.setup(800,600) circle = turtle.Turtle()...
作为一个程序员,CPU的这些硬核知识你必须会!
CPU对每个程序员来说,是个既熟悉又陌生的东西? 如果你只知道CPU是中央处理器的话,那可能对你并没有什么用,那么作为程序员的我们,必须要搞懂的就是CPU这家伙是如何运行的,尤其要搞懂它里面的寄存器是怎么一回事,因为这将让你从底层明白程序的运行机制。 随我一起,来好好认识下CPU这货吧 把CPU掰开来看 对于CPU来说,我们首先就要搞明白它是怎么回事,也就是它的内部构造,当然,CPU那么牛的一个东...
破14亿,Python分析我国存在哪些人口危机!
一、背景 二、爬取数据 三、数据分析 1、总人口 2、男女人口比例 3、人口城镇化 4、人口增长率 5、人口老化(抚养比) 6、各省人口 7、世界人口 四、遇到的问题 遇到的问题 1、数据分页,需要获取从1949-2018年数据,观察到有近20年参数:LAST20,由此推测获取近70年的参数可设置为:LAST70 2、2019年数据没有放上去,可以手动添加上去 3、将数据进行 行列转换 4、列名...
web前端javascript+jquery知识点总结
1.Javascript 语法.用途 javascript 在前端网页中占有非常重要的地位,可以用于验证表单,制作特效等功能,它是一种描述语言,也是一种基于对象(Object)和事件驱动并具有安全性的脚本语言 ...
Python实战:抓肺炎疫情实时数据,画2019-nCoV疫情地图
今天,群里白垩老师问如何用python画武汉肺炎疫情地图。白垩老师是研究海洋生态与地球生物的学者,国家重点实验室成员,于不惑之年学习python,实为我等学习楷模。先前我并没有关注武汉肺炎的具体数据,也没有画过类似的数据分布图。于是就拿了两个小时,专门研究了一下,遂成此文。
听说想当黑客的都玩过这个Monyer游戏(1~14攻略)
第零关 进入传送门开始第0关(游戏链接) 请点击链接进入第1关: 连接在左边→ ←连接在右边 看不到啊。。。。(只能看到一堆大佬做完的留名,也能看到菜鸡的我,在后面~~) 直接fn+f12吧 &lt;span&gt;连接在左边→&lt;/span&gt; &lt;a href="first.php"&gt;&lt;/a&gt; &lt;span&gt;←连接在右边&lt;/span&gt; o...
在家远程办公效率低?那你一定要收好这个「在家办公」神器!
相信大家都已经收到国务院延长春节假期的消息,接下来,在家远程办公可能将会持续一段时间。 但是问题来了。远程办公不是人在电脑前就当坐班了,相反,对于沟通效率,文件协作,以及信息安全都有着极高的要求。有着非常多的挑战,比如: 1在异地互相不见面的会议上,如何提高沟通效率? 2文件之间的来往反馈如何做到及时性?如何保证信息安全? 3如何规划安排每天工作,以及如何进行成果验收? ...... ...
作为一个程序员,内存和磁盘的这些事情,你不得不知道啊!!!
截止目前,我已经分享了如下几篇文章: 一个程序在计算机中是如何运行的?超级干货!!! 作为一个程序员,CPU的这些硬核知识你必须会! 作为一个程序员,内存的这些硬核知识你必须懂! 这些知识可以说是我们之前都不太重视的基础知识,可能大家在上大学的时候都学习过了,但是嘞,当时由于老师讲解的没那么有趣,又加上这些知识本身就比较枯燥,所以嘞,大家当初几乎等于没学。 再说啦,学习这些,也看不出来有什么用啊!...
渗透测试-灰鸽子远控木马
木马概述 灰鸽子( Huigezi),原本该软件适用于公司和家庭管理,其功能十分强大,不但能监视摄像头、键盘记录、监控桌面、文件操作等。还提供了黑客专用功能,如:伪装系统图标、随意更换启动项名称和表述、随意更换端口、运行后自删除、毫无提示安装等,并采用反弹链接这种缺陷设计,使得使用者拥有最高权限,一经破解即无法控制。最终导致被黑客恶意使用。原作者的灰鸽子被定义为是一款集多种控制方式于一体的木马程序...
Python:爬取疫情每日数据
前言 目前每天各大平台,如腾讯、今日头条都会更新疫情每日数据,他们的数据源都是一样的,主要都是通过各地的卫健委官网通报。 以全国、湖北和上海为例,分别为以下三个网站: 国家卫健委官网:http://www.nhc.gov.cn/xcs/yqtb/list_gzbd.shtml 湖北卫健委官网:http://wjw.hubei.gov.cn/bmdt/ztzl/fkxxgzbdgrfyyq/xxfb...
这个世界上人真的分三六九等,你信吗?
偶然间,在知乎上看到一个问题 一时间,勾起了我深深的回忆。 以前在厂里打过两次工,做过家教,干过辅导班,做过中介。零下几度的晚上,贴过广告,满脸、满手地长冻疮。 再回首那段岁月,虽然苦,但让我学会了坚持和忍耐。让我明白了,在这个世界上,无论环境多么的恶劣,只要心存希望,星星之火,亦可燎原。 下文是原回答,希望能对你能有所启发。 如果我说,这个世界上人真的分三六九等,...
B 站上有哪些很好的学习资源?
哇说起B站,在小九眼里就是宝藏般的存在,放年假宅在家时一天刷6、7个小时不在话下,更别提今年的跨年晚会,我简直是跪着看完的!! 最早大家聚在在B站是为了追番,再后来我在上面刷欧美新歌和漂亮小姐姐的舞蹈视频,最近两年我和周围的朋友们已经把B站当作学习教室了,而且学习成本还免费,真是个励志的好平台ヽ(.◕ฺˇд ˇ◕ฺ;)ノ 下面我们就来盘点一下B站上优质的学习资源: 综合类 Oeasy: 综合...
雷火神山直播超两亿,Web播放器事件监听是怎么实现的?
Web播放器解决了在手机浏览器和PC浏览器上播放音视频数据的问题,让视音频内容可以不依赖用户安装App,就能进行播放以及在社交平台进行传播。在视频业务大数据平台中,播放数据的统计分析非常重要,所以Web播放器在使用过程中,需要对其内部的数据进行收集并上报至服务端,此时,就需要对发生在其内部的一些播放行为进行事件监听。 那么Web播放器事件监听是怎么实现的呢? 01 监听事件明细表 名...
3万字总结,Mysql优化之精髓
本文知识点较多,篇幅较长,请耐心学习 MySQL已经成为时下关系型数据库产品的中坚力量,备受互联网大厂的青睐,出门面试想进BAT,想拿高工资,不会点MySQL优化知识,拿offer的成功率会大大下降。 为什么要优化 系统的吞吐量瓶颈往往出现在数据库的访问速度上 随着应用程序的运行,数据库的中的数据会越来越多,处理时间会相应变慢 数据是存放在磁盘上的,读写速度无法和内存相比 如何优化 设计...
Python新型冠状病毒疫情数据自动爬取+统计+发送报告+数据屏幕(三)发送篇
今天介绍的项目是使用 Itchat 发送统计报告 项目功能设计: 定时爬取疫情数据存入Mysql 进行数据分析制作疫情报告 使用itchat给亲人朋友发送分析报告 基于Django做数据屏幕 使用Tableau做数据分析 来看看最终效果 目前已经完成,预计2月12日前更新 使用 itchat 发送数据统计报告 itchat 是一个基于 web微信的一个框架,但微信官方并不允许使用这...
作为程序员的我,大学四年一直自学,全靠这些实用工具和学习网站!
我本人因为高中沉迷于爱情,导致学业荒废,后来高考,毫无疑问进入了一所普普通通的大学,实在惭愧???? 我又是那么好强,现在学历不行,没办法改变的事情了,所以,进入大学开始,我就下定决心,一定要让自己掌握更多的技能,尤其选择了计算机这个行业,一定要多学习技术。 在进入大学学习不久后,我就认清了一个现实:我这个大学的整体教学质量和学习风气,真的一言难尽,懂的人自然知道怎么回事? 怎么办?我该如何更好的提升自...
粒子群算法求解物流配送路线问题(python)
1.Matlab实现粒子群算法的程序代码:https://www.cnblogs.com/kexinxin/p/9858664.html matlab代码求解函数最优值:https://blog.csdn.net/zyqblog/article/details/80829043 讲解通俗易懂,有数学实例的博文:https://blog.csdn.net/daaikuaichuan/article/...
教你如何编写第一个简单的爬虫
很多人知道爬虫,也很想利用爬虫去爬取自己想要的数据,那么爬虫到底怎么用呢?今天就教大家编写一个简单的爬虫。 下面以爬取笔者的个人博客网站为例获取第一篇文章的标题名称,教大家学会一个简单的爬虫。 第一步:获取页面 #!/usr/bin/python # coding: utf-8 import requests #引入包requests link = "http://www.santostang....
前端JS初级面试题二 (。•ˇ‸ˇ•。)老铁们!快来瞧瞧自己都会了么
1. 传统事件绑定和符合W3C标准的事件绑定有什么区别? 传统事件绑定 &lt;div onclick=""&gt;123&lt;/div&gt; div1.onclick = function(){}; &lt;button onmouseover=""&gt;&lt;/button&gt; 注意: 如果给同一个元素绑定了两次或多次相同类型的事件,那么后面的绑定会覆盖前面的绑定 (不支持DOM事...
相关热词 c# 压缩图片好麻烦 c#计算数组中的平均值 c#获取路由参数 c#日期精确到分钟 c#自定义异常必须继承 c#查表并返回值 c# 动态 表达式树 c# 监控方法耗时 c# listbox c#chart显示滚动条
立即提问