H.264 Video Streaming System on Embedded Platform

H.264 tv set cyclosis scheme on infix PlatformABSTRACTThe adoption of technological products akin digital tv and television castigate conferencing has make bulge photo drift an ready research atomic snatch 18a.This address pre moves the consolidation of a tv set pennant mental faculty into a service line H.264/AVC convertr track a TMSDM6446EVM enter broadcast. The main objective of this estimate is to strive real- era floating of the standstillline H.264/AVC telecasting bothplace a local anaesthetic ara net hit (LAN) which is a interrupt of the supervision depiction musical ar placement.The en cryptograph of baseline H.264/AVC and the hardw be comp championnts of the platform ar premier converseed. Various be adrift conversations communications protocols ar studied in nightspot to natural process the boob tube exemplar on the DM6446 carte. The multi-threaded c everyplaceing en computer coder plan is design to encode in the al unneurotic word-painting frames into H.264/AVC format onto a wedge. For the television organization cyclosis, dissipate etymon Live555 MediaServer was apply to pour television clay schooling to a opposed VLC node oer LAN.Initially, star bear down teeming was apply from PC to PC. Upon success amply execution of instrument on PC, the flick bourge one(a)r was wayed to the come on. The steps complicated in porting the Live555 lotion were kindredwise depict in the encompass. Both unicast and multicast blame cyclosis were utilise in the television set teemer.Due to the problems of file teem, the brood cyclosis show up was adopted. Several mannerologies were discussed in incorpo point the movie streamer and the encoder weapons platform. Modification was do twain the encoder computer programme and the Live555 operation to overlord bed cyclosis of H.264/AVC telly. Results of twain file and live streaming leave be shown in this report . The follow throughed exposure streamer module give be apply as a base module of the telly surveillance system.Chapter 1 groundwork 1.1. Background world-shaking breakthroughs have been made everyplace the last few years in the argona of digital depiction contraction technologies. As such(prenominal)(prenominal) applications devising use of these technologies have in addition become popular and continue to be of active research trespassics today. For recitation, digital television and photo conferencing ar near of the applications that atomic name 18 now commsolely en deliberateered in our everyday lives. One application of pastime here is to make use of the technologies to action a film television camera surveillance system which alonet end raise the security of consumers descent and home environment.In typical surveillance systems, the captured exposure is sent everywhere a cable earningss to be monitored and stored at remote stations. As the captu red raw(prenominal) characterisation contains large amount of m whizzy of selective training, it will be of advantage to prime(prenominal) iron out the selective information by using a muscular contraction technique in advance it is transferred oer the mesh. One such compression technique that is sufficient for this vitrine of application is the H.264 cryptogram mensuration.H.264 mark is stop than the another(prenominal) coding technique for television pass catcher streaming as it is much than than robust to information passing acceptes and coding efficiency, which ar authorized factors when streaming is performed all everywhere a sh bed Local field of insure interlock. As at that place is an increasing word meaning of H.264 coding and the availability of amply computing provide infix systems, digital impression surveillance system based on H.264 on insert platform is in that locationfore a feasible and a potentially more cost- in force(p) system.Implementing a H.264 picture show streaming system on an embedded platform is a logical elongation of video surveillance systems which ar still typical implemented using spirited computing provide stations (e.g. PC). In a embedded var., a digital business lineize mastercessor (DSP) forms the core of the embedded system and executes the intense signal touch on algorithmic program. Current embedded systems typical in like manner accept ne 2rk features which change the carrying stunned of info streaming applications. To facilitate information streaming, a outlet of net profit protocol meters have also universe define, and are presently employ for digital video applications.1.2. Objective and kitchen stoveThe objective of this last year stick forth is to implement a video surveillance system based on the H.264 coding standard running on an embedded platform. Such a system contains extensive oscilloscopes of functionalities and would consider exte nsive amount of training time if implemented from scratch. Hence this frame is to focus on the entropy streaming feel of a video surveillance system. afterward some initial investigation and experimentation, it is decided to recoil the main scope of the plan to developing a live streaming H.264 based video system running on a DM6446 EVM development platform. The breakdown of the work to be progressive performed are past identify as follows1. Familiarization of open base live555 streaming media waiterDue to the complexity of implementing the variant standard protocols needed for multimedia streaming, the live555 media emcee program is use as a base to implement the streaming of the H.264.based video selective information.2. blow of stored H.264 file oer the networkThe live555 is hence modified to support streaming of raw encoded H.264 file from the DM6446 EVM instrument panel oer the network. Knowledge of H.264 coding standard is necessary in order to parse the file stream in the beginning streaming everywhere the network.3. Modifying a demonstrate version of an encoder program and integrating it to irritateher with live555 to extend to live streamingThe demo encoder was modified to get off encoded video info to the Live555 program which would do the necessary sheafization to be streamed over the network. Since entropy is passed from one process to another, various inter-process communication techniques were studied and used in this project.1.3. Re link upence worksThe re inaugurations used for this project are as follows1. DM6446 (DaVinci) Evaluation module2. SWANN C500 Professional CCTV photographic camera Solution 400 TV Lines CCD Color television camera3. LCD demonstrate4. IR Remote Control5. TI Davinci demo version of MontaVista Linux Pro v4.06. A Personal Workstation with Centos v5.07. VLC p mold v.0.9.8a as leaf node8. Open germ live555 program (downloaded from www.live555.com)The system setup of this project is shown to a lower place1.4. Report memorial tabletThis report consists of 7 chapters.Chapter 1 introduces the motivation tush embedded video streaming system and defines the scope of the project.Chapter 2 illustrates the video belles-lettres critique of the H.264/AVC video coding technique and the various streaming protocols which are to be implemented in the project.Chapter 3 explains the hardware literature review of the platform being used in the project. The computer architecture, memory management, inter-process communication and the software product excessivelyls are also discussed in this chapter.Chapter 4 explains the execution of the encoder program of the DM6446EVM board. The interaction of the various threads in this multi-threaded application is also discussed to fully go steady the encoder program.Chapter 5 gives an overview of the Live555 MediaServer which is used as a base to implement the video streamer module on the board. Adding support to unicast and multicast stream ing, porting of live555 to the board and receiving video stream on remote VCL lymph gland are explained in this chapter.Chapter 6 explains the confinements of file streaming and moving towards live streaming system. Various integration systemologies and modification to both encoder program and live555 program are shown as rise up.Chapters 7 summarize the implementation de ingredients of file and live streaming, analysis the performance of these results.Chapter 8 gives the conclusion by stating the current limitation and problems, scope for future implementation.Chapter 2 picture lit Review 2.1. H.264/AVC Video Codec OverviewH.264 is the close advanced and in vogue(p) video coding technique. Although in that location are numerous video coding schemes like H.26x and MPEG, H.264/AVC made many improvements and tools for coding efficiency and mis take resiliency. This chapter short will discuss the network aspect of the video coding technique. It will also carry error resili ency needed for transmittal of video data over the network. For a more detailed explanation of the H.264/AVC, refer to appendix A.2.1.1. Network Abstraction Layer (NAL)The aim of the NAL is to get a line that the data coming from the VCL layer is network worthy so that the data sort of a little be used for numerous systems. NAL facilitates the mapping of H.264/AVC VCL data for incompatible communicate layers such as* RTP/IP real-time streaming over pumped up(p) and wire slight mediums* antithetical storage file formats such as MP4, MMS, AVI and etceteraThe judgments of NAL and error boldness techniques of the H.264/AVC will be discussed in the by-line separate of the report. NAL UnitsThe encoded data from the VCL are packed into NAL unit of measurements. A NAL unit represents a parcel of land which makes up of a authorized subjugate of bytes. The first byte of the NAL unit is called the forefront byte which specifys the data type of the NAL unit. The remaining byte s make up the payload data of the NAL unit.The NAL unit bodily anatomical structure allows provision for different lift systems videlicet computer software-oriented and turning stream-oriented. To cater for flake stream-oriented send off systems like MPEG-2, the NAL units are organized into byte stream format. These units are prefixed by a specific start code prefix of one- troika bytes which is namely 0x000001. The start code prefix reads and the start of distributively NAL units and thitherof delineate the boundaries of the units.For parcel-oriented transport systems, the encoded video data are transported via piece of lands delineate by transport protocols. Hence, the boundaries of the NAL units are known with pop having to include start code prefix byte. The details of packetization of NAL units will be discussed in later sections of the report.NAL units are except categorise into two types* VCL unit comprises of encoded video data Non-VCL unit comprises of ad ditional information like logical argument sets which is the strategic school principal information. withal contains supplementary sweetener information (SEI) which contains the timing information and other data which increases the usability of the decoded video signal. annoy unitsA group of NAL units which adhere to a sure form is called a access unit. When one access unit is decoded, one decoded picture is formed. In the table 1 below, the functions of the NAL units derived from the access units are explained. info/Error hardihood techniquesH.264/AVC has several(prenominal)(prenominal)(prenominal) techniques to mitigate error/data loss which is an essential gauge when it comes to streaming applications. The techniques are as follows literary argument sets contains information that is being utilize to large number of VCL NAL units. It comprises of two kinds of parameter sets rank Parameter set (SPS) learning pertaining to grade of encoded picture Picture Parameter Set (PPS) Information pertaining to one or more individualist picturesThe preceding(prenominal) mentioned parameters hardly changes and hence it need not be transmitted repeatedly and saves overhead. The parameter sets trick be sent in-band which is carried in the same channel as the VCL NAL units. It drive out also be sent out-of-band using undeviating transport protocol. in that locationfore, it enhances the resiliency towards data and error loss. Flexible Macroblock Ordering (FMO)FMO maps the macroblocks to different slice groups. In the event of any slice group loss, lacking data is dissemble up by interpolating from the other slice groups. pleonasm Slices (RS)Redundant federal agency of the picture after federal agency be stored in the special slices. If the loss of the airplane pilot slice occurs, the decoder can make use of the redundant slices to determine the original slice.These techniques introduced in the H.264/AVC makes the codec more robust and alive towar ds data and error loss.2.1.2. Profiles and LevelsA profile of a codec is defined as the set of features place to meet a certain(p) specifications of intended applications For the H.264/AVC codec, it is defined as a set of features identify to generate a conformist spotlight stream. A level is imposes restrictions on some mention parameters of the function stream.In H.264/AVC, there are three profiles namely Baseline, Main and Extended. 5 shows the relationship mingled with these profiles. The Baseline profile is most likely to be used by network cameras and encoders as it requires limited computing re seeds. It is quite ideal to make use of this profile to support real-time streaming applications in a embedded platform.2.2. Overview of Video driftIn introductory systems, accessing video data across network exploit the download and cope with approach. In this approach, the knob had to wait until the full-page video data is downloaded to the media role bumper before black market out begins. To combat the long initial play out persist, the concept of streaming was introduced.Streaming allows the lymph node to play out the earlier part of the video data whilst still transferring the remaining part of the video data. The study advantage of the streaming concept is that the video data need not be stored in the thickenings computer as compared to the traditionalistic download and play approach. This reduces the long initial play out appreciation undergo by the invitee.Streaming adopts the traditional client/server mannerl. The client connects to the listening server and ask for video data. The server organizes video data over to the client for play out of video data.2.2.1. Types of Streaming there are three different types of streaming video data. They are pre-recorded/ file streaming, live/real-time streaming and synergetic streaming.* Pre-recorded/live streaming The encoded video is stored into a file and the system streams the file over th e network. A major overhead is that there is a long initial play out check over (10-15s) experienced by the client.* Live/real-time streaming The encoded video is streamed over the network straight without being stored into a file. The initial play out delay reduces. love must be taken to ensure that play out rate does not exceed displace rate which whitethorn result in jerky the picture. On the other hand, if the displace rate is too slow, the packets arriving at the client may be dropped, causation in a freezing the picture. The timing requirement for the end-to-end delay is more compressed in this scenario.* Interactive streaming Like live streaming, the video is streamed straight over the network. It responds to users retain input such as rewind, ruin, stop, play and onwards the special(prenominal) video stream. The system should respond in accordance to those inputs by the user.In this project, both pre-recorded and live streaming are implemented. just about func tionality of interactive streaming visualizes like stop and play are also part of the system.2.2.2. Video Streaming System modulesVideo commencementThe intent of the video source is to capture the raw video sequence. The CCTV camera is used as the video source in this project. Most cameras are of analogue inputs and these inputs are connected to the encoding station via video connections. This project makes use of solo one video source due to the limitation of the video connections on the encoding station. The raw video sequence is and then passed onto the encoding station. encoding StationThe aim of the encoding station digitized and encodes the raw video sequence into the desired format. In the actual system, the encoding is do by the DM6446 board into the H.264/AVC format. Since the hardware encoding is CPU intensive, this forms the bottleneck of the whole streaming system. The H.264 video is passed onto the video streamer server module of the system.Video Streaming and Web ServerThe intent of the video streaming server is to packetize the H.264/AVC to be streamed over the network. It serves the craves from individual clients. It necessitate to support the total bandwidth requirements of the particular video stream requested by clients. WebServer offers a uniform resource locator link which connects to the video streaming server. For this project, the video streaming server module is embedded inside DM6446 board and it is serves every individual clients requests.Video PlayerThe video player acts a client connecting to and requesting video data from the video streaming server. in one case the video data is posed, the video player buffers the data for a while and then begins play out of data. The video player used for this project is the VideoLAN (VLC) Player. It has the germane(predicate) H.264/AVC codec so that it can decode and play the H264/AVC video data.2.2.3. Unicast VS MulticastThere are two key delivery techniques employed by streaming med ia distribution.Unicast transmittance system is the direct of data to one particular network destination armament over a packet switched network. It establishes two way point-to-point connection between client and server. The client communicates directly with the server via this connection. The drawback is that every connection receives a separate video stream which uses up network bandwidth rapidly.Multicast transmission is the sending of only one copy of data via the network so that many clients can receive simultaneously. In video streaming, it is more cost effective to send single copy of video data over the network so as to conserve the network bandwidth. Since multicast is not connection oriented, the clients cannot fancy the streams that they can receive.In this project, unicast transmission is used to stream encoded video over the network. The client connects directly to the DM6446 board where it gets the encoded video data. The project can well be increase to multica st transmission.2.3. Streaming communications protocolsWhen streaming video content over a network, a number of network protocols are used. These protocols are well defined by the mesh Engineering Task Force (IETF) and the Internet Society (IS) and authenticated in Request for Comments (RFC) documents. These standards are adopted by many developers today.In this project, the same standards are also employed in order to prosperously stream H.264/AVC content over a easy Local Area Network (LAN). The following sections will discuss active the various protocols that are studied in the course of this project.2.3.1. Real- meter Streaming Protocol (RTSP)The most commonly used application layer protocol is RTSP. RTSP acts a dominance protocol to media streaming servers. It establishes connection between two end points of the system and control media academic academic school terms. Clients issue VCR-like commands like play and respite to facilitate the control of real-time playbac k of media streams from the servers. However, this protocol is not involved in the transport of the media stream over the network. For this project, RTSP version 1.0 is used.RTSP StatesLike the Hyper Text carry-over Protocol (HTTP), it contains several methods. They are OPTIONS, DESCRIBE, SETUP, recreate, PAUSE, RECORD and TEARDOWN. These commands are sent by using the RTSP URL. The nonremittal port number used in this protocol is 554. An slip of such as URL is rtsp// OPTIONS An OPTIONS request returns the types of request that the server will accept. An example of the request isOPTIONS rtsp//155.69.148.136554/test.264 RTSP/1.0CSeq 1rnUser- means VLC media PlayerThe CSeq parameter keeps track of the number of request send to the server and it is incremented every time a hot request is issued. The User-agent refers to the client making the request.* DESCRIBE This method gets the presentation or the media object identified in the request URL from the server. An example of such a requestDESCRIBE rtsp//155.69.148.138554/test.264 RTSP/1.0CSeq 2rn take up application/sdprnUser agent VLC media PlayerThe Accept psyche is used to describe the formats understood by the client. All the low-level formatting of the media resource must be present in the DESCRIBE method that it describes. SETUP This method will specify the mode of transport appliance to be used for the media stream. A typical example isSETUP rtsp//155.69.148.138554/test.264 RTSP/1.0CSeq 3rn reassign RTP/AVP unicast client_port = 1200-1201User agent VLC media PlayerThe charge drift specifies the transport tool to be used. In this case, real-time transport protocol is used in a unicast manner. The relevant client port number is also reflected and it is selected randomly by the server. Since RTSP is a stateful protocol, a sitting is created upon successful acknowledgement to this method. PLAY This method request the server to start sending the data via the transport mechanism stated in the SETUP met hod. The URL is the same as the other methods except for school term 6 float npt= 0.000- rnThe academic term header specifies the unique session id. This is important as server may establish various sessions and this keep tracks of them. The Range header puzzles play time to the beginning and plays till the end of the shop.* PAUSE This method informs the server to pause sending of the media stream. Once the PAUSE request is sent, the range header will capture the position at which the media stream is paused. When a PLAY request is sent again, the client will resume compete from the current position of the media stream as specified in the range header.RSTP Status CodesWhenever the client sends a request pass to the server, the server forms a equivalent chemical reaction message to be sent to the client. The response codes are similar to HTTP as they are both in ASCII text. They are as follows200 OK301 Redirection405 Method non Allowed451 Parameter Not still454 Session Not Fou nd457 Invalid Range461 Unsupported convey462 Destination unreachedThese are some of the RTSP status codes. There are many others but the codes mentioned above are of importance in the scene of this project.2.3.2. Real-time Transport Protocol (RTP)RTP is a defined packet structure which is used for transporting media stream over the network. It is a transport layer protocol but developers view it as a application layer protocol stack. This protocol facilitates jitter salary and ferret oution of incorrect sequence arrival of data which is common for transmission over IP network. For the transmission of media data over the network, it is important that packets fuck off in a timely manner as it is loss genial but not delay tolerant. Due to the high latency of contagious disease Control Protocol in establishing connections, RTP is often built on top of the User Datagram Protocol (UDP). RTP also supports multicast transmission of data.RTP is also a stateful protocol as a session is established before data can be packed into the RTP packet and sent over the network. The session contains the IP address of the destination and port number of the RTP which is normally an even number. The following section will explain around the packet structure of RTP which is used for transmission.RTP Packet StructureThe below shows a RTP packet header which is appended in front of the media data.sThe minimum size of the RTP header is 12 bytes.. Optional extension information may be present after the header information. The dish out of the header are V (2 bits) to indicate the version number of the protocol. stochastic variable used in this project is 2. P (Padding) (1 bit) to indicate if there hyperbolise which can be used for encryption algorithm X (Extension) (1 bit) to indicate if there is extension information between header and payload data. CC (CSRC Count) (4 bits) indicates the number of CSRC identifiers M (Marker) (1 bit) used by application to indicate data has specific relevance in the perspective of the application. The mount for M bit marks the end of video data in this project PT (Payload Type) (7 bits) to indicate the type of payload data carried by the packet. H.264 is used for this project Sequence number (16 bits) incremented by one for every RTP packet. It is used to detect packet loss and out of sequence packet arrival. Based on this information, application can take appropriate action to correct them. Time Stamp (32 bits) receivers use this information to play samples at correct intervals of time. from each one stream has independent time stamps. SSRC (32 bits) it unique identifies source of the stream. CSRC sources of a stream from different sources are enumerated according to its source IDs.This project does not involve the use of Extension field in the packet header and hence will not be explained in this report. Once this header information is appended to the payload data, the packet is sent over the network to the client to be played. The table below summarizes the payload types of RTP and highlighted region is of interest in this project. fudge 2 Payload Types of RTP Packets2.3.3. RTP Control Protocol (RTCP)RTCP is a sister protocol which is used in conjunction with the RTP. It provides out-of-band statistical and control information to the RTP session. This provides certain Quality of Service (QoS) for transmission of video data over the network.The aboriginal functions of the RTCP are* To gather statistical information about the quality aspect of the media stream during a RTP session. This data is sent to the session media source and its participants. The source can exploit this information for adaptive media encoding and detect transmission errors.* It provides canonical end point identifiers (CNAME) to all its session participants. It allows unique assignment of end points across different application instances and serves as a third party monitoring tool.* It also sends RTCP reports to all i ts session participants. By doing so, the calling bandwidth increases proportionally. In order to avoid congestion, RTCP has bandwidth management techniques to only use 5% of the total session bandwidth.RTCP statistical data is sent odd numbered ports. For instance, if RTP port number is 196, then RTCP will use the 197 as its port number. There is no default port number depute to RTCP.RTCP Message TypesRTCP sends several types of packets different from RTP packets. They are sender report, receiver report, source description and bye. transmitter Report (SR) Sent periodically by senders to report the transmission and answer statistics of RTP packets sent in a period of time. It also includes the senders SSRC and senders packet count information. The timestamp of the RTP packet is also sent to allow the receiver to synchronize the RTP packets. The bandwidth demand for SR is 25% of RTCP bandwidth. Receiver Report (RR) It reports the QoS to other receivers and senders. Information l ike highest sequence number received, inter arrival jitter of RTP packets and subdivision of packets loss further explains the QoS of the transmitted media streams. The bandwidth required for RR is 75% of the RTCP bandwidth. Source rendering (SDES) Sends the CNAME to its session participants. superfluous information like name, address of the owner of the source can also be sent. End of Participation (BYE) The source sends a BYE message to indicate that it is shutting down the stream. It serves as an declaration that a particular end point is leaving the conference.Further RTCP ConsiderationThis protocol is important to ensure that QoS standards are achieved. The acceptable frequencies of these reports are less than one minute. In major application, the frequency may increase as RTCP bandwidth control mechanism. Then, the statistical reporting on the quality of the media stream becomes inaccurate.Since there are no long delays introduced between the reports in this project, the R TCP is adopted to integrated a certain level of QoS on streaming H.264/AVC video over embedded platform.2.3.4. Session Description Protocol (SDP)The Session Description Protocol is a standard to describe streaming media initialization parameters. These initializations describe the sessions for session announcement, session invitation and parameter negotiation. This protocol can be used together with RTSP. In the preceding(prenominal) sections of this chapter, SDP is used in the DESCRIBE state of RTSP to get sessions media initialization parameters. SDP is scalable to include different media types and formats.SDP SyntaxThe session is set forth by evaluate/value pairs. The syntax of SDP are summarized in the below.In this project, the use of SDP is important in streaming as the client is VLC Media Player. If the streaming is done via RTSP, then VLC expects a sdp description from the server in order to setup the session and facilitate the playback of the streaming media.Chapter 3 H ardware literary works Review 3.1. first appearance to Texas Instrument DM6446EVM DavinciTMThe development of this project based on the DM6446EVM board. It is necessary to understand the hardware and software aspects of this board. The DM6446 board has a weapon system processor in operation(p) at a clock look sharp up to 300MHz and a C64x Digital Signal Processor operating at a clock speed of up to 600MHz.3.1.1. Key Features of DM6446The key features that are shown in the above are* 1 video port which supports composite of S video* 4 video DAC outputs component, RGB, composite* 256 MB of DDR2 DRAM* UART, Media instrument panel interface (SD, xD, SM, MS ,MMC Cards)* 16 MB of non-volatile Flash Memory, 64 MB NAND Flash, 4 MB SRAM* USB2 interface* 10/100 MBS Ethernet interface* Configurable hit load options* IR Remote Interface, real time clock via MSP4303.1.2. DM6446EVM architectureThe architecture of the DM6446 board is organized into several subsystems. By knowing the archite cture of the DM6446, the developer can then design and built his application module on the boards underlining architecture.The shows that DM6446 has three subsystems which are connected to the underlying hardware peripherals. This provides a decoupled architecture which allows the developers to implement his applications on a particular subsystem without having to modify the other subsystems. Some of subsystems are discussed in the next sections.ARM SubsystemThe ARM subsystem is responsible for the master control of the DM6446 board. It handles the system-level initializations, configurations, user interface, connectivity functions and control of DSP subsystems. The ARM has a larger program memory space and better context switching capabilities and hence it is more suited to handle complex and multi tasks of the system.DSP SubsystemThe DSP subsystem is mainly the encoding the raw captured video frames into the desired format. It performs several number crunching operations in order to achieve the desired compression technique. It works together with the Video Imaging Coprocessor to compress the video frames.Video Imaging Coprocessor (VICP)The VICP is a signal processing library which contains various software algorithms that execute on VICP hardware accelerator. It helps the DSP by victorious over enumeration of varied intensive tasks. Since hardware implementation of number cruH.264 Video Streaming System on Embedded PlatformH.264 Video Streaming System on Embedded PlatformABSTRACTThe adoption of technological products like digital television and video conferencing has made video streaming an active research area.This report presents the integration of a video streamer module into a baseline H.264/AVC encoder running a TMSDM6446EVM embedded platform. The main objective of this project is to achieve real-time streaming of the baseline H.264/AVC video over a local area network (LAN) which is a part of the surveillance video system.The encoding of baseline H .264/AVC and the hardware components of the platform are first discussed. Various streaming protocols are studied in order to implement the video streamer on the DM6446 board. The multi-threaded application encoder program is used to encode raw video frames into H.264/AVC format onto a file. For the video streaming, open source Live555 MediaServer was used to stream video data to a remote VLC client over LAN.Initially, file streaming was implemented from PC to PC. Upon successfully implementation on PC, the video streamer was ported to the board. The steps involved in porting the Live555 application were also described in the report. Both unicast and multicast file streaming were implemented in the video streamer.Due to the problems of file streaming, the live streaming approach was adopted. Several methodologies were discussed in integrating the video streamer and the encoder program. Modification was made both the encoder program and the Live555 application to achieve live streami ng of H.264/AVC video. Results of both file and live streaming will be shown in this report. The implemented video streamer module will be used as a base module of the video surveillance system.Chapter 1 Introduction 1.1. BackgroundSignificant breakthroughs have been made over the last few years in the area of digital video compression technologies. As such applications making use of these technologies have also become prevalent and continue to be of active research topics today. For example, digital television and video conferencing are some of the applications that are now commonly encountered in our daily lives. One application of interest here is to make use of the technologies to implement a video camera surveillance system which can enhance the security of consumers business and home environment.In typical surveillance systems, the captured video is sent over a cable networks to be monitored and stored at remote stations. As the captured raw video contains large amount of data , it will be of advantage to first compress the data by using a compression technique before it is transferred over the network. One such compression technique that is suitable for this type of application is the H.264 coding standard.H.264 coding is better than the other coding technique for video streaming as it is more robust to data losses and coding efficiency, which are important factors when streaming is performed over a shared Local Area Network. As there is an increasing acceptance of H.264 coding and the availability of high computing power embedded systems, digital video surveillance system based on H.264 on embedded platform is hence a feasible and a potentially more cost-effective system.Implementing a H.264 video streaming system on an embedded platform is a logical extension of video surveillance systems which are still typical implemented using high computing power stations (e.g. PC). In a embedded version, a Digital Signal Processor (DSP) forms the core of the embed ded system and executes the intensive signal processing algorithm. Current embedded systems typical also include network features which enable the implementation of data streaming applications. To facilitate data streaming, a number of network protocol standards have also being defined, and are currently used for digital video applications.1.2. Objective and ScopeThe objective of this final year project is to implement a video surveillance system based on the H.264 coding standard running on an embedded platform. Such a system contains extensive scopes of functionalities and would require extensive amount of development time if implemented from scratch. Hence this project is to focus on the data streaming aspect of a video surveillance system.After some initial investigation and experimentation, it is decided to confine the main scope of the project to developing a live streaming H.264 based video system running on a DM6446 EVM development platform. The breakdown of the work to be p rogressive performed are then identified as follows1. Familiarization of open source live555 streaming media serverDue to the complexity of implementing the various standard protocols needed for multimedia streaming, the live555 media server program is used as a base to implement the streaming of the H.264.based video data.2. Streaming of stored H.264 file over the networkThe live555 is then modified to support streaming of raw encoded H.264 file from the DM6446 EVM board over the network. Knowledge of H.264 coding standard is necessary in order to parse the file stream before streaming over the network.3. Modifying a demo version of an encoder program and integrating it together with live555 to achieve live streamingThe demo encoder was modified to send encoded video data to the Live555 program which would do the necessary packetization to be streamed over the network. Since data is passed from one process to another, various inter-process communication techniques were studied and used in this project.1.3. ResourcesThe resources used for this project are as follows1. DM6446 (DaVinci) Evaluation Module2. SWANN C500 Professional CCTV Camera Solution 400 TV Lines CCD Color Camera3. LCD Display4. IR Remote Control5. TI Davinci demo version of MontaVista Linux Pro v4.06. A Personal Workstation with Centos v5.07. VLC player v.0.9.8a as client8. Open source live555 program (downloaded from www.live555.com)The system setup of this project is shown below1.4. Report OrganizationThis report consists of 7 chapters.Chapter 1 introduces the motivation behind embedded video streaming system and defines the scope of the project.Chapter 2 illustrates the video literature review of the H.264/AVC video coding technique and the various streaming protocols which are to be implemented in the project.Chapter 3 explains the hardware literature review of the platform being used in the project. The architecture, memory management, inter-process communication and the software tools are also discussed in this chapter.Chapter 4 explains the execution of the encoder program of the DM6446EVM board. The interaction of the various threads in this multi-threaded application is also discussed to fully understand the encoder program.Chapter 5 gives an overview of the Live555 MediaServer which is used as a base to implement the video streamer module on the board. Adding support to unicast and multicast streaming, porting of live555 to the board and receiving video stream on remote VCL client are explained in this chapter.Chapter 6 explains the limitations of file streaming and moving towards live streaming system. Various integration methodologies and modification to both encoder program and live555 program are shown as well.Chapters 7 summarize the implementation results of file and live streaming, analysis the performance of these results.Chapter 8 gives the conclusion by stating the current limitation and problems, scope for future implementation.Chapter 2 Video Literat ure Review 2.1. H.264/AVC Video Codec OverviewH.264 is the most advanced and latest video coding technique. Although there are many video coding schemes like H.26x and MPEG, H.264/AVC made many improvements and tools for coding efficiency and error resiliency. This chapter briefly will discuss the network aspect of the video coding technique. It will also cover error resiliency needed for transmission of video data over the network. For a more detailed explanation of the H.264/AVC, refer to appendix A.2.1.1. Network Abstraction Layer (NAL)The aim of the NAL is to ensure that the data coming from the VCL layer is network worthy so that the data can be used for numerous systems. NAL facilitates the mapping of H.264/AVC VCL data for different transport layers such as* RTP/IP real-time streaming over wired and wireless mediums* Different storage file formats such as MP4, MMS, AVI and etc.The concepts of NAL and error robustness techniques of the H.264/AVC will be discussed in the follow ing parts of the report. NAL UnitsThe encoded data from the VCL are packed into NAL units. A NAL unit represents a packet which makes up of a certain number of bytes. The first byte of the NAL unit is called the header byte which indicates the data type of the NAL unit. The remaining bytes make up the payload data of the NAL unit.The NAL unit structure allows provision for different transport systems namely packet-oriented and bit stream-oriented. To cater for bit stream-oriented transport systems like MPEG-2, the NAL units are organized into byte stream format. These units are prefixed by a specific start code prefix of three bytes which is namely 0x000001. The start code prefix indicates and the start of each NAL units and hence defining the boundaries of the units.For packet-oriented transport systems, the encoded video data are transported via packets defined by transport protocols. Hence, the boundaries of the NAL units are known without having to include start code prefix byte . The details of packetization of NAL units will be discussed in later sections of the report.NAL units are further categorized into two types* VCL unit comprises of encoded video data Non-VCL unit comprises of additional information like parameter sets which is the important header information. Also contains supplementary enhancement information (SEI) which contains the timing information and other data which increases the usability of the decoded video signal. Access unitsA group of NAL units which adhere to a certain form is called a access unit. When one access unit is decoded, one decoded picture is formed. In the table 1 below, the functions of the NAL units derived from the access units are explained.Data/Error robustness techniquesH.264/AVC has several techniques to mitigate error/data loss which is an essential quality when it comes to streaming applications. The techniques are as follows Parameter sets contains information that is being applied to large number of VCL NAL u nits. It comprises of two kinds of parameter sets Sequence Parameter set (SPS) Information pertaining to sequence of encoded picture Picture Parameter Set (PPS) Information pertaining to one or more individual picturesThe above mentioned parameters hardly changes and hence it need not be transmitted repeatedly and saves overhead. The parameter sets can be sent in-band which is carried in the same channel as the VCL NAL units. It can also be sent out-of-band using reliable transport protocol. Therefore, it enhances the resiliency towards data and error loss. Flexible Macroblock Ordering (FMO)FMO maps the macroblocks to different slice groups. In the event of any slice group loss, missing data is masked up by interpolating from the other slice groups. Redundancy Slices (RS)Redundant representation of the picture can be stored in the redundant slices. If the loss of the original slice occurs, the decoder can make use of the redundant slices to recover the original slice.These techniq ues introduced in the H.264/AVC makes the codec more robust and resilient towards data and error loss.2.1.2. Profiles and LevelsA profile of a codec is defined as the set of features identified to meet a certain specifications of intended applications For the H.264/AVC codec, it is defined as a set of features identified to generate a conforming bit stream. A level is imposes restrictions on some key parameters of the bit stream.In H.264/AVC, there are three profiles namely Baseline, Main and Extended. 5 shows the relationship between these profiles. The Baseline profile is most likely to be used by network cameras and encoders as it requires limited computing resources. It is quite ideal to make use of this profile to support real-time streaming applications in a embedded platform.2.2. Overview of Video StreamingIn previous systems, accessing video data across network exploit the download and play approach. In this approach, the client had to wait until the whole video data is down loaded to the media player before play out begins. To combat the long initial play out delay, the concept of streaming was introduced.Streaming allows the client to play out the earlier part of the video data whilst still transferring the remaining part of the video data. The major advantage of the streaming concept is that the video data need not be stored in the clients computer as compared to the traditional download and play approach. This reduces the long initial play out delay experienced by the client.Streaming adopts the traditional client/server model. The client connects to the listening server and request for video data. The server sends video data over to the client for play out of video data.2.2.1. Types of StreamingThere are three different types of streaming video data. They are pre-recorded/ file streaming, live/real-time streaming and interactive streaming.* Pre-recorded/live streaming The encoded video is stored into a file and the system streams the file over the network. A major overhead is that there is a long initial play out delay (10-15s) experienced by the client.* Live/real-time streaming The encoded video is streamed over the network directly without being stored into a file. The initial play out delay reduces. Consideration must be taken to ensure that play out rate does not exceed sending rate which may result in jerky the picture. On the other hand, if the sending rate is too slow, the packets arriving at the client may be dropped, causing in a freezing the picture. The timing requirement for the end-to-end delay is more stringent in this scenario.* Interactive streaming Like live streaming, the video is streamed directly over the network. It responds to users control input such as rewind, pause, stop, play and forward the particular video stream. The system should respond in accordance to those inputs by the user.In this project, both pre-recorded and live streaming are implemented. Some functionality of interactive streaming con trols like stop and play are also part of the system.2.2.2. Video Streaming System modulesVideo SourceThe intent of the video source is to capture the raw video sequence. The CCTV camera is used as the video source in this project. Most cameras are of analogue inputs and these inputs are connected to the encoding station via video connections. This project makes use of only one video source due to the limitation of the video connections on the encoding station. The raw video sequence is then passed onto the encoding station.Encoding StationThe aim of the encoding station digitized and encodes the raw video sequence into the desired format. In the actual system, the encoding is done by the DM6446 board into the H.264/AVC format. Since the hardware encoding is CPU intensive, this forms the bottleneck of the whole streaming system. The H.264 video is passed onto the video streamer server module of the system.Video Streaming and WebServerThe role of the video streaming server is to pack etize the H.264/AVC to be streamed over the network. It serves the requests from individual clients. It needs to support the total bandwidth requirements of the particular video stream requested by clients. WebServer offers a URL link which connects to the video streaming server. For this project, the video streaming server module is embedded inside DM6446 board and it is serves every individual clients requests.Video PlayerThe video player acts a client connecting to and requesting video data from the video streaming server. Once the video data is received, the video player buffers the data for a while and then begins play out of data. The video player used for this project is the VideoLAN (VLC) Player. It has the relevant H.264/AVC codec so that it can decode and play the H264/AVC video data.2.2.3. Unicast VS MulticastThere are two key delivery techniques employed by streaming media distribution.Unicast transmission is the sending of data to one particular network destination host over a packet switched network. It establishes two way point-to-point connection between client and server. The client communicates directly with the server via this connection. The drawback is that every connection receives a separate video stream which uses up network bandwidth rapidly.Multicast transmission is the sending of only one copy of data via the network so that many clients can receive simultaneously. In video streaming, it is more cost effective to send single copy of video data over the network so as to conserve the network bandwidth. Since multicast is not connection oriented, the clients cannot control the streams that they can receive.In this project, unicast transmission is used to stream encoded video over the network. The client connects directly to the DM6446 board where it gets the encoded video data. The project can easily be extended to multicast transmission.2.3. Streaming ProtocolsWhen streaming video content over a network, a number of network protocols a re used. These protocols are well defined by the Internet Engineering Task Force (IETF) and the Internet Society (IS) and documented in Request for Comments (RFC) documents. These standards are adopted by many developers today.In this project, the same standards are also employed in order to successfully stream H.264/AVC content over a simple Local Area Network (LAN). The following sections will discuss about the various protocols that are studied in the course of this project.2.3.1. Real-Time Streaming Protocol (RTSP)The most commonly used application layer protocol is RTSP. RTSP acts a control protocol to media streaming servers. It establishes connection between two end points of the system and control media sessions. Clients issue VCR-like commands like play and pause to facilitate the control of real-time playback of media streams from the servers. However, this protocol is not involved in the transport of the media stream over the network. For this project, RTSP version 1.0 is used.RTSP StatesLike the Hyper Text Transfer Protocol (HTTP), it contains several methods. They are OPTIONS, DESCRIBE, SETUP, PLAY, PAUSE, RECORD and TEARDOWN. These commands are sent by using the RTSP URL. The default port number used in this protocol is 554. An example of such as URL is rtsp// OPTIONS An OPTIONS request returns the types of request that the server will accept. An example of the request isOPTIONS rtsp//155.69.148.136554/test.264 RTSP/1.0CSeq 1rnUser-agent VLC media PlayerThe CSeq parameter keeps track of the number of request send to the server and it is incremented every time a new request is issued. The User-agent refers to the client making the request.* DESCRIBE This method gets the presentation or the media object identified in the request URL from the server. An example of such a requestDESCRIBE rtsp//155.69.148.138554/test.264 RTSP/1.0CSeq 2rnAccept application/sdprnUser agent VLC media PlayerThe Accept header is used to describe the formats understood by t he client. All the initialization of the media resource must be present in the DESCRIBE method that it describes. SETUP This method will specify the mode of transport mechanism to be used for the media stream. A typical example isSETUP rtsp//155.69.148.138554/test.264 RTSP/1.0CSeq 3rnTransport RTP/AVP unicast client_port = 1200-1201User agent VLC media PlayerThe Transport header specifies the transport mechanism to be used. In this case, real-time transport protocol is used in a unicast manner. The relevant client port number is also reflected and it is selected randomly by the server. Since RTSP is a stateful protocol, a session is created upon successful acknowledgement to this method. PLAY This method request the server to start sending the data via the transport mechanism stated in the SETUP method. The URL is the same as the other methods except forSession 6Range npt= 0.000- rnThe Session header specifies the unique session id. This is important as server may establish various sessions and this keep tracks of them. The Range header positions play time to the beginning and plays till the end of the range.* PAUSE This method informs the server to pause sending of the media stream. Once the PAUSE request is sent, the range header will capture the position at which the media stream is paused. When a PLAY request is sent again, the client will resume playing from the current position of the media stream as specified in the range header.RSTP Status CodesWhenever the client sends a request message to the server, the server forms a equivalent response message to be sent to the client. The response codes are similar to HTTP as they are both in ASCII text. They are as follows200 OK301 Redirection405 Method Not Allowed451 Parameter Not Understood454 Session Not Found457 Invalid Range461 Unsupported Transport462 Destination UnreachableThese are some of the RTSP status codes. There are many others but the codes mentioned above are of importance in the context of this project.2.3.2. Real-time Transport Protocol (RTP)RTP is a defined packet structure which is used for transporting media stream over the network. It is a transport layer protocol but developers view it as a application layer protocol stack. This protocol facilitates jitter compensation and detection of incorrect sequence arrival of data which is common for transmission over IP network. For the transmission of media data over the network, it is important that packets arrive in a timely manner as it is loss tolerant but not delay tolerant. Due to the high latency of Transmission Control Protocol in establishing connections, RTP is often built on top of the User Datagram Protocol (UDP). RTP also supports multicast transmission of data.RTP is also a stateful protocol as a session is established before data can be packed into the RTP packet and sent over the network. The session contains the IP address of the destination and port number of the RTP which is usually an even number. The foll owing section will explain about the packet structure of RTP which is used for transmission.RTP Packet StructureThe below shows a RTP packet header which is appended in front of the media data.sThe minimum size of the RTP header is 12 bytes.. Optional extension information may be present after the header information. The fields of the header are V (2 bits) to indicate the version number of the protocol. Version used in this project is 2. P (Padding) (1 bit) to indicate if there padding which can be used for encryption algorithm X (Extension) (1 bit) to indicate if there is extension information between header and payload data. CC (CSRC Count) (4 bits) indicates the number of CSRC identifiers M (Marker) (1 bit) used by application to indicate data has specific relevance in the perspective of the application. The setting for M bit marks the end of video data in this project PT (Payload Type) (7 bits) to indicate the type of payload data carried by the packet. H.264 is used for this p roject Sequence number (16 bits) incremented by one for every RTP packet. It is used to detect packet loss and out of sequence packet arrival. Based on this information, application can take appropriate action to correct them. Time Stamp (32 bits) receivers use this information to play samples at correct intervals of time. Each stream has independent time stamps. SSRC (32 bits) it unique identifies source of the stream. CSRC sources of a stream from different sources are enumerated according to its source IDs.This project does not involve the use of Extension field in the packet header and hence will not be explained in this report. Once this header information is appended to the payload data, the packet is sent over the network to the client to be played. The table below summarizes the payload types of RTP and highlighted region is of interest in this project.Table 2 Payload Types of RTP Packets2.3.3. RTP Control Protocol (RTCP)RTCP is a sister protocol which is used in conjunction with the RTP. It provides out-of-band statistical and control information to the RTP session. This provides certain Quality of Service (QoS) for transmission of video data over the network.The primary functions of the RTCP are* To gather statistical information about the quality aspect of the media stream during a RTP session. This data is sent to the session media source and its participants. The source can exploit this information for adaptive media encoding and detect transmission errors.* It provides canonical end point identifiers (CNAME) to all its session participants. It allows unique identification of end points across different application instances and serves as a third party monitoring tool.* It also sends RTCP reports to all its session participants. By doing so, the traffic bandwidth increases proportionally. In order to avoid congestion, RTCP has bandwidth management techniques to only use 5% of the total session bandwidth.RTCP statistical data is sent odd numbered p orts. For instance, if RTP port number is 196, then RTCP will use the 197 as its port number. There is no default port number assigned to RTCP.RTCP Message TypesRTCP sends several types of packets different from RTP packets. They are sender report, receiver report, source description and bye. Sender Report (SR) Sent periodically by senders to report the transmission and reception statistics of RTP packets sent in a period of time. It also includes the senders SSRC and senders packet count information. The timestamp of the RTP packet is also sent to allow the receiver to synchronize the RTP packets. The bandwidth required for SR is 25% of RTCP bandwidth. Receiver Report (RR) It reports the QoS to other receivers and senders. Information like highest sequence number received, inter arrival jitter of RTP packets and fraction of packets loss further explains the QoS of the transmitted media streams. The bandwidth required for RR is 75% of the RTCP bandwidth. Source Description (SDES) Se nds the CNAME to its session participants. Additional information like name, address of the owner of the source can also be sent. End of Participation (BYE) The source sends a BYE message to indicate that it is shutting down the stream. It serves as an announcement that a particular end point is leaving the conference.Further RTCP ConsiderationThis protocol is important to ensure that QoS standards are achieved. The acceptable frequencies of these reports are less than one minute. In major application, the frequency may increase as RTCP bandwidth control mechanism. Then, the statistical reporting on the quality of the media stream becomes inaccurate.Since there are no long delays introduced between the reports in this project, the RTCP is adopted to incorporate a certain level of QoS on streaming H.264/AVC video over embedded platform.2.3.4. Session Description Protocol (SDP)The Session Description Protocol is a standard to describe streaming media initialization parameters. These i nitializations describe the sessions for session announcement, session invitation and parameter negotiation. This protocol can be used together with RTSP. In the previous sections of this chapter, SDP is used in the DESCRIBE state of RTSP to get sessions media initialization parameters. SDP is scalable to include different media types and formats.SDP SyntaxThe session is described by attribute/value pairs. The syntax of SDP are summarized in the below.In this project, the use of SDP is important in streaming as the client is VLC Media Player. If the streaming is done via RTSP, then VLC expects a sdp description from the server in order to setup the session and facilitate the playback of the streaming media.Chapter 3 Hardware Literature Review 3.1. Introduction to Texas Instrument DM6446EVM DavinciTMThe development of this project based on the DM6446EVM board. It is necessary to understand the hardware and software aspects of this board. The DM6446 board has a ARM processor operating at a clock speed up to 300MHz and a C64x Digital Signal Processor operating at a clock speed of up to 600MHz.3.1.1. Key Features of DM6446The key features that are shown in the above are* 1 video port which supports composite of S video* 4 video DAC outputs component, RGB, composite* 256 MB of DDR2 DRAM* UART, Media Card interface (SD, xD, SM, MS ,MMC Cards)* 16 MB of non-volatile Flash Memory, 64 MB NAND Flash, 4 MB SRAM* USB2 interface* 10/100 MBS Ethernet interface* Configurable boot load options* IR Remote Interface, real time clock via MSP4303.1.2. DM6446EVM ArchitectureThe architecture of the DM6446 board is organized into several subsystems. By knowing the architecture of the DM6446, the developer can then design and built his application module on the boards underlining architecture.The shows that DM6446 has three subsystems which are connected to the underlying hardware peripherals. This provides a decoupled architecture which allows the developers to implement his applica tions on a particular subsystem without having to modify the other subsystems. Some of subsystems are discussed in the next sections.ARM SubsystemThe ARM subsystem is responsible for the master control of the DM6446 board. It handles the system-level initializations, configurations, user interface, connectivity functions and control of DSP subsystems. The ARM has a larger program memory space and better context switching capabilities and hence it is more suited to handle complex and multi tasks of the system.DSP SubsystemThe DSP subsystem is mainly the encoding the raw captured video frames into the desired format. It performs several number crunching operations in order to achieve the desired compression technique. It works together with the Video Imaging Coprocessor to compress the video frames.Video Imaging Coprocessor (VICP)The VICP is a signal processing library which contains various software algorithms that execute on VICP hardware accelerator. It helps the DSP by taking over computation of varied intensive tasks. Since hardware implementation of number cru