weixin_39617470
weixin_39617470
2020-12-02 06:43

Do not see the old volumes after performing storage vMotion of the vm

Steps: 1. Create a volume on sharedVmfs-0 datastore. VM is on shared-Vmfs-0 datastore. 2. Run a container with volume attached 3. While the container is running, migrate the vm to a different datastore - storage vMotion 4. Docker volume ls and Admin cli volume ls does not list the volume created before migration.

Logs have been attached at the end.

Steps and their output are as follows:

  1. Created a volume named testVolXXX

root-rdops-vm02-dhcp-52-237:~# docker volume create -d vsphere:latest --name testVolXXX
testVolXXX
root-rdops-vm02-dhcp-52-237:~#
root-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      testVolXXX-0
root-rdops-vm02-dhcp-52-237:~#
root-rdops-vm02-dhcp-52-237:~# docker volume inspect testVolXXX-0
[
    {
        "Driver": "vsphere:latest",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/testVolXXX-0",
        "Name": "testVolXXX-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Fri Jul 21 19:07:53 2017",
            "created by VM": "ubuntu-VM0.0",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        }
    }
]
root-rdops-vm02-dhcp-52-237:~#

  1. Admin cli volume ls

[root-rdops-vm04-dhcp-109-225:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume      Datastore     VMGroup   Capacity  Used  Filesystem  Policy  Disk Format  Attached-to   Access      Attach-as               Created By    Created Date
----------  ------------  --------  --------  ----  ----------  ------  -----------  ------------  ----------  ----------------------  ------------  ------------------------
testVolXXX  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         ubuntu-VM0.0  read-write  independent_persistent  ubuntu-VM0.0  Fri Jul 21 19:07:53 2017

  1. Ran a container with volume attached.

root-rdops-vm02-dhcp-52-237:~#  docker run -it --volume-driver=vsphere -v testVolXXX-0:/vol1 --name ub_XXX ubuntu
root:/#

  1. While container is running, migrated the vm to a sharedVmfs-1.

  2. Docker volume ls and admin cli volume ls does not lists any volumes.


root-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
root-rdops-vm02-dhcp-52-237:~#

[root-rdops-vm04-dhcp-109-225:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
Volume  Datastore  VMGroup  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access  Attach-as  Created By  Created Date
------  ---------  -------  --------  ----  ----------  ------  -----------  -----------  ------  ---------  ----------  ------------

[root-rdops-vm04-dhcp-109-225:~]

Logs: docker-volume-vsphere.txt vmdk_ops.txt

该提问来源于开源项目:vmware-archive/vsphere-storage-for-docker

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

9条回答

  • weixin_39559079 weixin_39559079 5月前

    Checked vmdk_ops.txt. After migration, we do receive another attach request from the same VM.

    
    07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] *** disk_attach: VMDK /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk to VM 'ubuntu-VM0.0' , bios uuid = 564d9ddb-5e8a-844b-4247-6a33944412b1, VC uuid=52693e9f-79d8-9b9a-0350-aa212f770b45)
    07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [WARNING] Failed to find VM by VC UUID 52693e9f-79d8-9b9a-0350-aa212f770b45, trying BIOS UUID 564d9ddb-5e8a-844b-4247-6a33944412b1
    07/21/17 19:10:45 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] Attaching /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk as independent_persistent
    07/21/17 19:10:46 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] Disk /vmfs/volumes/sharedVmfs-0/dockvols/_DEFAULT/testVolXXX.vmdk successfully attached. controller pci_slot_number=160, disk_slot=5
    07/21/17 19:10:46 120912 [ubuntu-VM0.0-sharedVmfs-0._DEFAULT.testVolXXX] [INFO   ] executeRequest 'attach' completed with ret={'Unit': '5', 'ControllerPciSlotNumber': '160'}
    07/21/17 19:10:53 236486 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
    07/21/17 19:10:54 236486 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
    07/21/17 19:11:00 120912 [MainThread] [INFO   ] Started new thread : 782454302464 with target <function execrequestthread at> and args (11, 235223, b'{"cmd":"list","details":{"Name":""},"version":"2"}')
    07/21/17 19:11:00 120912 [Thread-111] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
    07/21/17 19:11:01 120912 [ubuntu-VM0.0-nolock-list] [INFO   ] executeRequest 'list' completed with ret=[{'Name': 'testVolXXX-0', 'Attributes': {}}]   -----> here the list complete successfully.
    07/21/17 19:13:15 120912 [VMChangeListener] [INFO   ] VMChangeListener: VM was powered down and then deleted right away. Fault msg: The object 'vim.VirtualMachine:14' has already been deleted or has not been completely created    ---->here seems the VM migrate not finished yet.
    07/21/17 19:13:45 237208 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
    07/21/17 19:13:46 237208 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
    07/21/17 19:14:39 237357 [MainThread] [INFO   ] Connecting to the local Service Instance as 'dcui' 
    07/21/17 19:14:40 237357 [MainThread] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
    07/21/17 19:23:10 120912 [MainThread] [INFO   ] Started new thread : 782454302464 with target <function execrequestthread at> and args (13, 236673, b'{"cmd":"list","details":{"Name":""},"version":"2"}')
    07/21/17 19:23:10 120912 [Thread-112] [INFO   ] Auth DB /etc/vmware/vmdkops/auth-db is missing, allowing all access
    07/21/17 19:23:11 120912 [ubuntu-VM0.0-nolock-list] [INFO   ] executeRequest 'list' completed with ret=[]   -----> here list cmd return with empty
    </function></function>
    点赞 评论 复制链接分享
  • weixin_39559079 weixin_39559079 5月前

    Could you check in the datastore /vmfs/volumes/sharedVmfs-0, does the vmdk file for volume "testVolXXX" exist or not?

    点赞 评论 复制链接分享
  • weixin_39617470 weixin_39617470 5月前

    No, I do not see the VMDKs at /vmfs/volumes/sharedVmfs-0

    点赞 评论 复制链接分享
  • weixin_39559079 weixin_39559079 5月前

    I checked the Anup's setup (after migration). In "/vmfs/volumes/sharedVmfs-0/dockervol", the directory to store volumes for _DEFAULT vmgroup is missing.

    
    ls /vmfs/volumes/sharedVmfs-0/dockvols/ -al
    total 1040
    drwxr-xr-x    1 root     root           420 Jul 21 23:13 .
    drwxr-xr-t    1 root     root          2800 Jul 21 23:24 ..
    lrwxrwxrwx    1 root     root            72 Jul 21 22:26 _DEFAULT -> /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111
    
    

    The symbol link "_DEFAULT" is there, but the actual directory /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111 is missing.

    We need to see why "/vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111" and vmdk files under it is missing.

    点赞 评论 复制链接分享
  • weixin_39617470 weixin_39617470 5月前

    What's the behavior if the volume is attached and then svmotion is done?

    And if they volume is created like volume it should be accessible always. What happens if volume is created this way, then attached to the VM and then svmotion

    点赞 评论 复制链接分享
  • weixin_39617470 weixin_39617470 5月前

    The volume will be in the vm's folder

    点赞 评论 复制链接分享
  • weixin_39559079 weixin_39559079 5月前

    I did the following test 1. ubuntu-VM0.1 is in sharedVmfs-0, and I create a volume from this VM

    
    root-rdops-vm02-dhcp-52-237:~# docker volume create -d vsphere:latest --name testVolXXX-1
    testVolXXX-1
    root-rdops-vm02-dhcp-52-237:~# docker volume ls
    DRIVER              VOLUME NAME
    vsphere:latest      testVolXXX-1
    
    
    1. start a container to use this volume
    
    root-rdops-vm02-dhcp-52-237:~# docker run -it --volume-driver=vsphere -v testVolXXX-1:/vol1 --name ub_XXX ubuntu
    
    
    1. run svmotion to migrate ubuntu-VM0.1 from sharedVmfs-0 to sharedVmfs-1.
    2. After svmotion, the volume is still visible, but it also shows some volumes which stores VM info
    
    root-rdops-vm02-dhcp-52-237:~# docker volume ls
    DRIVER              VOLUME NAME
    vsphere:latest      testVolXXX-1
    vsphere:latest      ubuntu-VM0.1_1-1
    vsphere:latest      ubuntu-VM0.1_2-1
    vsphere:latest      ubuntu-VM0.1_4-1
    
    
    点赞 评论 复制链接分享
  • weixin_39559079 weixin_39559079 5月前

    Yes, you are right. I tried to repro based on the steps Anup metioned, the volume "testVolXXX" is migrated to datastore "shared-Vmfs-1" along with the vmdk file which represent the VM itself. The volume is renamed and put into the VM directory on "shared-Vmfs-1". So no volume in "/vmfs/volumes/shared-Vmfs-0/dockervols " and no volume in "/vmfs/volumes/shared-Vmfs-1/dockervols. That is why "vmdkops_admin volume ls" and "docker volume ls" shows no volume after svmotion.

    I think it is the expected behavior of svmotion, and we should document this in the known issue. Any comments?

    点赞 评论 复制链接分享
  • weixin_39897687 weixin_39897687 5月前

    we know svmotion messes up the locations and the names of attached vmdks, so yes, this is an expected behavior and yes. we should document it.

    点赞 评论 复制链接分享

相关推荐