Kinect 360 ubuntu 18.04 bionic ROS test

Teste Kinect 360 no ubuntu 18.04 bionic com ROS

Para ligar a Microsoft Kinect 360 ao computador é necessário ter um adaptador que consiste num transformador, um connector (onde vai ligar o cabo da Kinect 360) e um cabo com USB que sai da caixa do connector e vai ligar a uma porta USB no computador.

dav

Os testes foram efectuados num portatil Lenovo IdeaPad S145, com o Linux Ubuntu Bionic (18.04)

Informações sobre a distro

~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic

Informações sobre o kernel

~$ uname -a
Linux ideapad 5.4.0-66-generic #74~18.04.2-Ubuntu SMP Fri Feb 5 11:17:31 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Informções USB da Kinect 360

~$ lsusb
Bus 001 Device 010: ID 045e:02ae Microsoft Corp. Xbox NUI Camera
Bus 001 Device 008: ID 045e:02b0 Microsoft Corp. Xbox NUI Motor
Bus 001 Device 009: ID 045e:02ad Microsoft Corp. Xbox NUI Audio

 

Exploração da Kinect 360 no linux sem o ROS

Podemos explorar a kinect fora do ros com o pacote libfreenect-bin. Este pacote pode ser instalado usando o apt.

Instalação no ubuntu do libfreenect-bin, ou do freenect, que o inclui.

sudo apt install freenect

Depois de instalado podemos executar um programa de exploração da kinect 360.

~$ freenect-glview

Este comando apresenta uma janela com a imagem da deep camera do lado esquerdo e uma imagem da camara rgb do lado direito.

Além disso permite controlar alguns aspectos da kinect, nomeadamente o controlo da inclinação. Os comandos disponiveis são os seguintes:

‘w’ – tilt up, ‘s’ – level, ‘x’ – tilt down,
‘0’-‘6’ – select LED mode,
‘+’ & ‘-‘ – change IR intensity
‘f’ – change video format, ‘m’ – mirror video,
‘o’ – rotate video with accelerometer
‘e’ – auto exposure, ‘b’ – white balance,
‘r’ – raw color, ‘n’ – near mode (K4W only)

Saida na consola após a execução do comando

Kinect camera test
Number of devices found: 1
GL thread
[Stream 70] Negotiated packet size 1920
write_register: 0x0105 <= 0x00
write_register: 0x0006 <= 0x00
write_register: 0x0012 <= 0x03
write_register: 0x0013 <= 0x01
write_register: 0x0014 <= 0x1e
write_register: 0x0006 <= 0x02
write_register: 0x0017 <= 0x00
[Stream 80] Negotiated packet size 1920
write_register: 0x000c <= 0x00
write_register: 0x000d <= 0x01
write_register: 0x000e <= 0x1e
write_register: 0x0005 <= 0x01
[Stream 70] Lost 240 total packets in 0 frames (inf lppf)
[Stream 70] Lost 270 total packets in 0 frames (inf lppf)
write_register: 0x0047 <= 0x00
(...)

A exploração do audio pode ser efectuada com o comando:

freenect-micview

No entanto o teste, conforme se pode verificar nas mensagens exibidas abaixo, correu mal.

~$ freenect-micview
Number of devices found: 1
Trying to open ./audios.bin as firmware...
Trying to open /home/inaciose/.libfreenect/audios.bin as firmware...
Trying to open /usr/local/share/libfreenect/audios.bin as firmware...
Trying to open /usr/share/libfreenect/audios.bin as firmware...
Trying to open ./../Resources/audios.bin as firmware...
upload_firmware: failed to find firmware file.
upload_firmware failed: -2
This is the libfreenect microphone waveform viewer. Press 'q' to quit or spacebar to pause/unpause the view.
audio: invalid magic in iso IN packet: 00000000

Pelos vistos o problema é que não consegue encontrar o ficheiro com o firmware (audios.bin).

Uma pesquisa na web referiu a necessidade de executar o scritp fwfetcher.py para gerar o ficheiro audio.bin, que posteriormente deve ser copiado para uma das localizações acima.

Tentei resolver o assunto instalando o pacote python-freenect com o seguinte comando, mas não resolveu.

sudo apt install python-freenect

Terei que pesquisar melhor este assunto mais tarde.

 

Exploração da Kinect 360 no ROS

As potencialidades da kinect 360 podem ser exploradas no ros com uma coleção de pacotes que podem ser instalados com o apt.

Pelos vistos existem há pelo menos duas soluções para a integração da kinect 360 no ROS melodic, uma baseada no openni e outra no freenect.

  • openni
    http://wiki.ros.org/openni_camera
    http://wiki.ros.org/openni_launch
  • freenect-stack
    http://wiki.ros.org/freenect_stack
    ros-melodic-freenect-stack

Numa primeira parte descrevo a instalação e a exploração da solução openni, que era a que conhecia.

Kinect 360 com o ros openni no ubuntu bionic

A instalação do ros openni no ubuntu  pode ser feita com o apt.

sudo apt-get install ros-melodic-vision-opencv
sudo apt-get install ros-melodic-openni-launch
sudo apt-get install ros-melodic-openni-camera
sudo apt-get install ros-melodic-rgbd-launch

Nas minhas experiencias anteriores tinha instalado o pacote freenect para o ros, no entanto o pacote libfreenect para o ros melodic não está disponvel no ubunto bionic. pelo menos quando o tentamos instalar com o comando:

sudo apt install ros-melodic-libfreenect

Aparece o seguinte erro:

E: Unable to locate package ros-melodic-libfreenect

Pelos vistos, e conforme exibido na pagina do pacote, o pacote libfreenect está apenas disponivel para o ros Kinectic.

http://wiki.ros.org/libfreenect

A pesquisa dos pacotes disponiveis cujo nome inclui libfreenect revelou algumas opções

apt search libfreenect

As opções a considerar foram as seguintes.

ros-melodic-freenect-camera
ros-melodic-freenect-stack

No entanto a descrição ‘A libfreenect-based ROS driver for the Microsoft Kinect’ é exactamente nas duas o que não ajuda muito, mas ambos tem uma página que os descreve na wiki do ROS, que já referi acima, e nela descobri que o freenect-camera faz parte do freenect-stack.

Indeciso sobre se devia instalar o pacote libfreenect-stack, resolvi experimentar se o seguinte comando funcionava (e funcionou).

roslaunch openni_launch openni.launch

O comando devolve algumas informações sobre a kinect e publicando alguns topicos que podem ser usados no rviz

[ INFO] [1615640624.323407153]: Number devices connected: 1
[ INFO] [1615640624.323701914]: 1. device on bus 001:10 is a SensorV2 (2ae) from PrimeSense (45e) with serial id 'A00361A02199045A'
[ INFO] [1615640624.324962821]: Searching for device with index = 1
[ INFO] [1615640624.372737787]: Opened 'SensorV2' on bus 1:10 with serial number 'A00361A02199045A'
[ INFO] [1615640624.406604267]: rgb_frame_id = 'camera_rgb_optical_frame'
[ INFO] [1615640624.406676036]: depth_frame_id = 'camera_depth_optical_frame'
[ WARN] [1615640624.412769205]: Camera calibration file /home/inaciose/.ros/camera_info/rgb_A00361A02199045A.yaml not found.
[ WARN] [1615640624.412859410]: Using default parameters for RGB camera calibration.
[ WARN] [1615640624.412937672]: Camera calibration file /home/inaciose/.ros/camera_info/depth_A00361A02199045A.yaml not found.
[ WARN] [1615640624.412998888]: Using default parameters for IR camera calibration.

 

Kinect 360 com openni no rviz

A exploração da kinect 360 no rviz pode ser efectuada recorrendo aos seguintes displays:

  • Image
  • Camera
  • Pointcloud2
  • DeepCloud

Os dois ultimos displays, têm configurações para atribuir cores aos pontos, sobreposição de imagem com a camera rgb, etc.

ros-melodic-freenect-stack
Numa primeira parte descrevo a instalação e a exploração da solução openni, que era a que conhecia.

Kinect 360 com o ros freenect-stack no ubuntu bionic

Nesta segunda parte descrevo a instalação e a exploração da soluçãofreenect-stack, que no decorrer deste processo me apercebi que é uma alternativa a solução baseada em openni.

A instalação do software necessário no ros é efectuada com o apt

sudo ros-melodic-freenect-stack

Decidi não instalar neste computador. Eventualmente vou tentar a instalação mais tarde no ros noetic.

Apendice

Tópicos publicados pelos programas lançados pelo comando:

roslaunch openni_launch openni.launch

Lista de tópicos

~$ rostopic list
/camera/depth/camera_info
/camera/depth/disparity
/camera/depth/image
/camera/depth/image/compressed
/camera/depth/image/compressed/parameter_descriptions
/camera/depth/image/compressed/parameter_updates
/camera/depth/image/compressedDepth
/camera/depth/image/compressedDepth/parameter_descriptions
/camera/depth/image/compressedDepth/parameter_updates
/camera/depth/image/theora
/camera/depth/image/theora/parameter_descriptions
/camera/depth/image/theora/parameter_updates
/camera/depth/image_raw
/camera/depth/image_raw/compressed
/camera/depth/image_raw/compressed/parameter_descriptions
/camera/depth/image_raw/compressed/parameter_updates
/camera/depth/image_raw/compressedDepth
/camera/depth/image_raw/compressedDepth/parameter_descriptions
/camera/depth/image_raw/compressedDepth/parameter_updates
/camera/depth/image_raw/theora
/camera/depth/image_raw/theora/parameter_descriptions
/camera/depth/image_raw/theora/parameter_updates
/camera/depth/image_rect
/camera/depth/image_rect/compressed
/camera/depth/image_rect/compressed/parameter_descriptions
/camera/depth/image_rect/compressed/parameter_updates
/camera/depth/image_rect/compressedDepth
/camera/depth/image_rect/compressedDepth/parameter_descriptions
/camera/depth/image_rect/compressedDepth/parameter_updates
/camera/depth/image_rect/theora
/camera/depth/image_rect/theora/parameter_descriptions
/camera/depth/image_rect/theora/parameter_updates
/camera/depth/image_rect_raw
/camera/depth/image_rect_raw/compressed
/camera/depth/image_rect_raw/compressed/parameter_descriptions
/camera/depth/image_rect_raw/compressed/parameter_updates
/camera/depth/image_rect_raw/compressedDepth
/camera/depth/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth/image_rect_raw/compressedDepth/parameter_updates
/camera/depth/image_rect_raw/theora
/camera/depth/image_rect_raw/theora/parameter_descriptions
/camera/depth/image_rect_raw/theora/parameter_updates
/camera/depth/points
/camera/depth_rectify_depth/parameter_descriptions
/camera/depth_rectify_depth/parameter_updates
/camera/depth_registered/camera_info
/camera/depth_registered/disparity
/camera/depth_registered/hw_registered/image_rect
/camera/depth_registered/hw_registered/image_rect/compressed
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect/compressedDepth
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect/theora
/camera/depth_registered/hw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw
/camera/depth_registered/hw_registered/image_rect_raw/compressed
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/hw_registered/image_rect_raw/theora
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/hw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered/image
/camera/depth_registered/image/compressed
/camera/depth_registered/image/compressed/parameter_descriptions
/camera/depth_registered/image/compressed/parameter_updates
/camera/depth_registered/image/compressedDepth
/camera/depth_registered/image/compressedDepth/parameter_descriptions
/camera/depth_registered/image/compressedDepth/parameter_updates
/camera/depth_registered/image/theora
/camera/depth_registered/image/theora/parameter_descriptions
/camera/depth_registered/image/theora/parameter_updates
/camera/depth_registered/image_raw
/camera/depth_registered/image_raw/compressed
/camera/depth_registered/image_raw/compressed/parameter_descriptions
/camera/depth_registered/image_raw/compressed/parameter_updates
/camera/depth_registered/image_raw/compressedDepth
/camera/depth_registered/image_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/image_raw/compressedDepth/parameter_updates
/camera/depth_registered/image_raw/theora
/camera/depth_registered/image_raw/theora/parameter_descriptions
/camera/depth_registered/image_raw/theora/parameter_updates
/camera/depth_registered/points
/camera/depth_registered/sw_registered/camera_info
/camera/depth_registered/sw_registered/image_rect
/camera/depth_registered/sw_registered/image_rect/compressed
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect/compressedDepth
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect/theora
/camera/depth_registered/sw_registered/image_rect/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect/theora/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw
/camera/depth_registered/sw_registered/image_rect_raw/compressed
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressed/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/compressedDepth/parameter_updates
/camera/depth_registered/sw_registered/image_rect_raw/theora
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_descriptions
/camera/depth_registered/sw_registered/image_rect_raw/theora/parameter_updates
/camera/depth_registered_rectify_depth/parameter_descriptions
/camera/depth_registered_rectify_depth/parameter_updates
/camera/driver/parameter_descriptions
/camera/driver/parameter_updates
/camera/ir/camera_info
/camera/ir/image_raw
/camera/ir/image_raw/compressed
/camera/ir/image_raw/compressed/parameter_descriptions
/camera/ir/image_raw/compressed/parameter_updates
/camera/ir/image_raw/compressedDepth
/camera/ir/image_raw/compressedDepth/parameter_descriptions
/camera/ir/image_raw/compressedDepth/parameter_updates
/camera/ir/image_raw/theora
/camera/ir/image_raw/theora/parameter_descriptions
/camera/ir/image_raw/theora/parameter_updates
/camera/ir/image_rect_ir
/camera/ir/image_rect_ir/compressed
/camera/ir/image_rect_ir/compressed/parameter_descriptions
/camera/ir/image_rect_ir/compressed/parameter_updates
/camera/ir/image_rect_ir/compressedDepth
/camera/ir/image_rect_ir/compressedDepth/parameter_descriptions
/camera/ir/image_rect_ir/compressedDepth/parameter_updates
/camera/ir/image_rect_ir/theora
/camera/ir/image_rect_ir/theora/parameter_descriptions
/camera/ir/image_rect_ir/theora/parameter_updates
/camera/ir_rectify_ir/parameter_descriptions
/camera/ir_rectify_ir/parameter_updates
/camera/projector/camera_info
/camera/rgb/camera_info
/camera/rgb/image_color
/camera/rgb/image_color/compressed
/camera/rgb/image_color/compressed/parameter_descriptions
/camera/rgb/image_color/compressed/parameter_updates
/camera/rgb/image_color/compressedDepth
/camera/rgb/image_color/compressedDepth/parameter_descriptions
/camera/rgb/image_color/compressedDepth/parameter_updates
/camera/rgb/image_color/theora
/camera/rgb/image_color/theora/parameter_descriptions
/camera/rgb/image_color/theora/parameter_updates
/camera/rgb/image_mono
/camera/rgb/image_mono/compressed
/camera/rgb/image_mono/compressed/parameter_descriptions
/camera/rgb/image_mono/compressed/parameter_updates
/camera/rgb/image_mono/compressedDepth
/camera/rgb/image_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_mono/compressedDepth/parameter_updates
/camera/rgb/image_mono/theora
/camera/rgb/image_mono/theora/parameter_descriptions
/camera/rgb/image_mono/theora/parameter_updates
/camera/rgb/image_raw
/camera/rgb/image_raw/compressed
/camera/rgb/image_raw/compressed/parameter_descriptions
/camera/rgb/image_raw/compressed/parameter_updates
/camera/rgb/image_raw/compressedDepth
/camera/rgb/image_raw/compressedDepth/parameter_descriptions
/camera/rgb/image_raw/compressedDepth/parameter_updates
/camera/rgb/image_raw/theora
/camera/rgb/image_raw/theora/parameter_descriptions
/camera/rgb/image_raw/theora/parameter_updates
/camera/rgb/image_rect_color
/camera/rgb/image_rect_color/compressed
/camera/rgb/image_rect_color/compressed/parameter_descriptions
/camera/rgb/image_rect_color/compressed/parameter_updates
/camera/rgb/image_rect_color/compressedDepth
/camera/rgb/image_rect_color/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_color/compressedDepth/parameter_updates
/camera/rgb/image_rect_color/theora
/camera/rgb/image_rect_color/theora/parameter_descriptions
/camera/rgb/image_rect_color/theora/parameter_updates
/camera/rgb/image_rect_mono
/camera/rgb/image_rect_mono/compressed
/camera/rgb/image_rect_mono/compressed/parameter_descriptions
/camera/rgb/image_rect_mono/compressed/parameter_updates
/camera/rgb/image_rect_mono/compressedDepth
/camera/rgb/image_rect_mono/compressedDepth/parameter_descriptions
/camera/rgb/image_rect_mono/compressedDepth/parameter_updates
/camera/rgb/image_rect_mono/theora
/camera/rgb/image_rect_mono/theora/parameter_descriptions
/camera/rgb/image_rect_mono/theora/parameter_updates
/camera/rgb_debayer/parameter_descriptions
/camera/rgb_debayer/parameter_updates
/camera/rgb_rectify_color/parameter_descriptions
/camera/rgb_rectify_color/parameter_updates
/camera/rgb_rectify_mono/parameter_descriptions
/camera/rgb_rectify_mono/parameter_updates
/rosout
/rosout_agg
/tf
/tf_static

 

Instructions for kinect 360 in ros noethic

Give your next Robot 3D vision: Kinect V1 with ROS Noetic