In this paper a system is designed on an FPGA using a Nios II soft-core processor, to detect the colour of a specific surface and moving a robot arm accordingly. The surface being detected is bounded by a starting mark and an ending mark, to define the region of interest. The surface is also divided into sections as rows and columns and each section can have any colour. Such a system has so many uses like for example warehouses or even in stores where their storing areas can be divided to sections and each section is coloured and a robot arm collects objects from these sections according to the section’s colour also the robot arm can organize objects in sections according to the section’s colour.