Skip to content
Snippets Groups Projects
Commit 4f13db3c authored by Nicolas Noble's avatar Nicolas Noble Committed by GitHub
Browse files

Merge pull request #8184 from nicolasnoble/nanopb-code-drop

Nanopb code drop
parents dff32e0b 47c0053b
Branches
Tags
No related merge requests found
Showing
with 6556 additions and 4 deletions
...@@ -14,6 +14,3 @@ ...@@ -14,6 +14,3 @@
[submodule "third_party/boringssl"] [submodule "third_party/boringssl"]
path = third_party/boringssl path = third_party/boringssl
url = https://github.com/google/boringssl.git url = https://github.com/google/boringssl.git
[submodule "third_party/nanopb"]
path = third_party/nanopb
url = https://github.com/nanopb/nanopb.git
Subproject commit f8ac463766281625ad710900479130c7fcb4d63b
*.gcda
*.gcno
*.gcov
*.o
*.pb.c
*.pb.h
*.pb
*.pyc
*_pb2.py
*~
*.tar.gz
.sconsign.dblite
config.log
.sconf_temp
tests/build
julkaisu.txt
dist
docs/*.html
docs/generator_flow.png
examples/simple/simple
examples/network_server/client
examples/network_server/server
examples/using_double_on_avr/decode_double
examples/using_double_on_avr/encode_double
examples/using_double_on_avr/test_conversions
examples/using_union_messages/decode
examples/using_union_messages/encode
generator/nanopb_pb2.pyc
# Travis CI has no ability to handle 3 langauges (c, c++, python)
# and it overrides $CC/$CXX if language is set to c/c++ (only one, not both).
#
# Set language to python since at least the result of that is something useful.
language: python
python:
- "2.7"
- "3.4"
# Manage the C/C++ compiler manually
env:
- CC=gcc CXX=g++
- CC=gcc-4.8 CXX=g++-4.8
- CC=gcc-4.9 CXX=g++-4.9
- CC=gcc-5 CXX=g++-5
- CC=clang CXX=clang++
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- gcc-4.8
- g++-4.8
- gcc-4.9
- g++-4.9
- gcc-5
- g++-5
before_install:
- export PATH=$HOME/.local/bin:$HOME/protobuf/bin:$PATH
- export MAKEFLAGS=-j$(grep processor /proc/cpuinfo | wc -l)
- $CC --version
- $CXX --version
- python --version
- lsb_release -a
# Seems to be issues with concurrent builds
#cache:
# directories:
# - $HOME/protobuf
install:
- curl -L https://github.com/google/protobuf/releases/download/v3.0.0-beta-1/protobuf-python-3.0.0-alpha-4.tar.gz | tar xzf -
&& pushd protobuf-3.0.0-alpha-4
&& ./configure --prefix=$HOME/protobuf && make && make install
&& pushd python && python setup.py build && python setup.py install && popd
&& popd
script:
- pushd generator/proto && make && popd
- pushd tests && python2 $(which scons) CC=$CC CXX=$CXX && popd
nanopb-0.3.4 (2015-09-26)
Fix handling of unsigned 8- and 16-bit enums (issue 164)
Fix generator on systems where python = python3. (issue 155)
Fix compiler warning on GCC 5.x (issue 171)
Make the generator better handle imported .protos (issue 165)
Add packed_enum option to generator.
Add syntax= line to .proto files (issue 167)
Add PlatformIO registry manifest file. (pr 156)
nanopb-0.3.3 (2015-04-10)
Fix missing files in Linux binary package (issue 146)
Fix generator bug when oneof is first field in a message. (issue 142)
Fix generator error when long_names:false is combined with Oneofs. (issue 147)
Fix oneof submessage initialization bug. (issue 149)
Fix problem with plugin options on Python 2.7.2 and older. (issue 153)
Fix crash when callback is inside oneof field. (issue 148)
Switch to .tar.gz format for Mac OS X packages. (issue 154)
Always define enum long names so that cross-file references work. (issue 118)
Add msgid generator option. (issue 151)
Improve comment support in .options files. (issue 145)
Updates for the CMake rule file, add cmake example.
Better error messages for syntax errors in .options file
nanopb-0.3.2 (2015-01-24)
Fix memory leaks with PB_ENABLE_MALLOC with some submessage hierarchies (issue 138)
Implement support for oneofs (C unions). (issues 131, 141)
Add int_size option for generator (issue 139)
Add compilation option to disable struct packing. (issue 136)
Change PB_RETURN_ERROR() macro to avoid compiler warnings (issue 140)
Fix build problems with protoc 3.0.0
Add support for POINTER type in extensions
Initialize also extension fields to defaults in pb_decode().
Detect too large varint values when decoding.
nanopb-0.3.1 (2014-09-11)
Fix security issue due to size_t overflows. (issue 132)
Fix memory leak with duplicated fields and PB_ENABLE_MALLOC
Fix crash if pb_release() is called twice.
Fix cyclic message support (issue 130)
Fix error in generated initializers for repeated pointer fields.
Improve tests (issues 113, 126)
nanopb-0.3.0 (2014-08-26)
NOTE: See docs/migration.html or online at
http://koti.kapsi.fi/~jpa/nanopb/docs/migration.html
for changes in this version. Most importantly, you need to add
pb_common.c to the list of files to compile.
Separated field iterator logic to pb_common.c (issue 128)
Change the _count fields to use pb_size_t datatype (issue 82)
Added PB_ prefix to macro names (issue 106)
Added #if version guard to generated files (issue 129)
Added migration document
nanopb-0.2.9 (2014-08-09)
NOTE: If you are using the -e option with the generator, you have
to prepend . to the argument to get the same behaviour as before.
Do not automatically add a dot with generator -e option. (issue 122)
Fix problem with .options file and extension fields. (issue 125)
Don't use SIZE_MAX macro, as it is not in C89. (issue 120)
Generate #defines for initializing message structures. (issue 79)
Add skip_message option to generator. (issue 121)
Add PB_PACKED_STRUCT support for Keil MDK-ARM toolchain (issue 119)
Give better messages about the .options file path. (issue 124)
Improved tests
nanopb-0.2.8 (2014-05-20)
Fix security issue with PB_ENABLE_MALLOC. (issue 117)
Add option to not add timestamps to .pb.h and .pb.c preambles. (issue 115)
Documentation updates
Improved tests
nanopb-0.2.7 (2014-04-07)
Fix bug with default values for extension fields (issue 111)
Fix some MISRA-C warnings (issue 91)
Implemented optional malloc() support (issue 80)
Changed pointer-type bytes field datatype
Add a "found" field to pb_extension_t (issue 112)
Add convenience function pb_get_encoded_size() (issue 16)
nanopb-0.2.6 (2014-02-15)
Fix generator error with bytes callback fields (issue 99)
Fix warnings about large integer constants (issue 102)
Add comments to where STATIC_ASSERT is used (issue 96)
Add warning about unknown field names on .options (issue 105)
Move descriptor.proto to google/protobuf subdirectory (issue 104)
Improved tests
nanopb-0.2.5 (2014-01-01)
Fix a bug with encoding negative values in int32 fields (issue 97)
Create binary packages of the generator + dependencies (issue 47)
Add support for pointer-type fields to the encoder (part of issue 80)
Fixed path in FindNanopb.cmake (issue 94)
Improved tests
nanopb-0.2.4 (2013-11-07)
Remove the deprecated NANOPB_INTERNALS functions from public API.
Document the security model.
Check array and bytes max sizes when encoding (issue 90)
Add #defines for maximum encoded message size (issue 89)
Add #define tags for extension fields (issue 93)
Fix MISRA C violations (issue 91)
Clean up pb_field_t definition with typedefs.
nanopb-0.2.3 (2013-09-18)
Improve compatibility by removing ternary operator from initializations (issue 88)
Fix build error on Visual C++ (issue 84, patch by Markus Schwarzenberg)
Don't stop on unsupported extension fields (issue 83)
Add an example pb_syshdr.h file for non-C99 compilers
Reorganize tests and examples into subfolders (issue 63)
Switch from Makefiles to scons for building the tests
Make the tests buildable on Windows
nanopb-0.2.2 (2013-08-18)
Add support for extension fields (issue 17)
Fix unknown fields in empty message (issue 78)
Include the field tags in the generated .pb.h file.
Add pb_decode_delimited and pb_encode_delimited wrapper functions (issue 74)
Add a section in top of pb.h for changing compilation settings (issue 76)
Documentation improvements (issues 12, 77 and others)
Improved tests
nanopb-0.2.1 (2013-04-14)
NOTE: The default callback function signature has changed.
If you don't want to update your code, define PB_OLD_CALLBACK_STYLE.
Change the callback function to use void** (issue 69)
Add support for defining the nanopb options in a separate file (issue 12)
Add support for packed structs in IAR and MSVC (in addition to GCC) (issue 66)
Implement error message support for the encoder side (issue 7)
Handle unterminated strings when encoding (issue 68)
Fix bug with empty strings in repeated string callbacks (issue 73)
Fix regression in 0.2.0 with optional callback fields (issue 70)
Fix bugs with empty message types (issues 64, 65)
Fix some compiler warnings on clang (issue 67)
Some portability improvements (issues 60, 62)
Various new generator options
Improved tests
nanopb-0.2.0 (2013-03-02)
NOTE: This release requires you to regenerate all .pb.c
files. Files generated by older versions will not
compile anymore.
Reformat generated .pb.c files using macros (issue 58)
Rename PB_HTYPE_ARRAY -> PB_HTYPE_REPEATED
Separate PB_HTYPE to PB_ATYPE and PB_HTYPE
Move STATIC_ASSERTs to .pb.c file
Added CMake file (by Pavel Ilin)
Add option to give file extension to generator (by Michael Haberler)
Documentation updates
nanopb-0.1.9 (2013-02-13)
Fixed error message bugs (issues 52, 56)
Sanitize #ifndef filename (issue 50)
Performance improvements
Add compile-time option PB_BUFFER_ONLY
Add Java package name to nanopb.proto
Check for sizeof(double) == 8 (issue 54)
Added generator option to ignore some fields. (issue 51)
Added generator option to make message structs packed. (issue 49)
Add more test cases.
nanopb-0.1.8 (2012-12-13)
Fix bugs in the enum short names introduced in 0.1.7 (issues 42, 43)
Fix STATIC_ASSERT macro when using multiple .proto files. (issue 41)
Fix missing initialization of istream.errmsg
Make tests/Makefile work for non-gcc compilers (issue 40)
nanopb-0.1.7 (2012-11-11)
Remove "skip" mode from pb_istream_t callbacks. Example implementation had a bug. (issue 37)
Add option to use shorter names for enum values (issue 38)
Improve options support in generator (issues 12, 30)
Add nanopb version number to generated files (issue 36)
Add extern "C" to generated headers (issue 35)
Add names for structs to allow forward declaration (issue 39)
Add buffer size check in example (issue 34)
Fix build warnings on MS compilers (issue 33)
nanopb-0.1.6 (2012-09-02)
Reorganize the field decoder interface (issue 2)
Improve performance in submessage decoding (issue 28)
Implement error messages in the decoder side (issue 7)
Extended testcases (alltypes test is now complete).
Fix some compiler warnings (issues 25, 26, 27, 32).
nanopb-0.1.5 (2012-08-04)
Fix bug in decoder with packed arrays (issue 23).
Extended testcases.
Fix some compiler warnings.
nanopb-0.1.4 (2012-07-05)
Add compile-time options for easy-to-use >255 field support.
Improve the detection of missing required fields.
Added example on how to handle union messages.
Fix generator error with .proto without messages.
Fix problems that stopped the code from compiling with some compilers.
Fix some compiler warnings.
nanopb-0.1.3 (2012-06-12)
Refactor the field encoder interface.
Improve generator error messages (issue 5)
Add descriptor.proto into the #include exclusion list
Fix some compiler warnings.
nanopb-0.1.2 (2012-02-15)
Make the generator to generate include for other .proto files (issue 4).
Fixed generator not working on Windows (issue 3)
nanopb-0.1.1 (2012-01-14)
Fixed bug in encoder with 'bytes' fields (issue 1).
Fixed a bug in the generator that caused a compiler error on sfixed32 and sfixed64 fields.
Extended testcases.
nanopb-0.1.0 (2012-01-06)
First stable release.
Copyright (c) 2011 Petteri Aimonen <jpa at nanopb.mail.kapsi.fi>
This software is provided 'as-is', without any express or
implied warranty. In no event will the authors be held liable
for any damages arising from the use of this software.
Permission is granted to anyone to use this software for any
purpose, including commercial applications, and to alter it and
redistribute it freely, subject to the following restrictions:
1. The origin of this software must not be misrepresented; you
must not claim that you wrote the original software. If you use
this software in a product, an acknowledgment in the product
documentation would be appreciated but is not required.
2. Altered source versions must be plainly marked as such, and
must not be misrepresented as being the original software.
3. This notice may not be removed or altered from any source
distribution.
Nanopb - Protocol Buffers for Embedded Systems
==============================================
[![Build Status](https://travis-ci.org/nanopb/nanopb.svg?branch=master)](https://travis-ci.org/nanopb/nanopb)
Nanopb is a small code-size Protocol Buffers implementation in ansi C. It is
especially suitable for use in microcontrollers, but fits any memory
restricted system.
* **Homepage:** http://kapsi.fi/~jpa/nanopb/
* **Documentation:** http://kapsi.fi/~jpa/nanopb/docs/
* **Downloads:** http://kapsi.fi/~jpa/nanopb/download/
* **Forum:** https://groups.google.com/forum/#!forum/nanopb
Using the nanopb library
------------------------
To use the nanopb library, you need to do two things:
1. Compile your .proto files for nanopb, using protoc.
2. Include pb_encode.c and pb_decode.c in your project.
The easiest way to get started is to study the project in "examples/simple".
It contains a Makefile, which should work directly under most Linux systems.
However, for any other kind of build system, see the manual steps in
README.txt in that folder.
Using the Protocol Buffers compiler (protoc)
--------------------------------------------
The nanopb generator is implemented as a plugin for the Google's own protoc
compiler. This has the advantage that there is no need to reimplement the
basic parsing of .proto files. However, it does mean that you need the
Google's protobuf library in order to run the generator.
If you have downloaded a binary package for nanopb (either Windows, Linux or
Mac OS X version), the 'protoc' binary is included in the 'generator-bin'
folder. In this case, you are ready to go. Simply run this command:
generator-bin/protoc --nanopb_out=. myprotocol.proto
However, if you are using a git checkout or a plain source distribution, you
need to provide your own version of protoc and the Google's protobuf library.
On Linux, the necessary packages are protobuf-compiler and python-protobuf.
On Windows, you can either build Google's protobuf library from source or use
one of the binary distributions of it. In either case, if you use a separate
protoc, you need to manually give the path to nanopb generator:
protoc --plugin=protoc-gen-nanopb=nanopb/generator/protoc-gen-nanopb ...
Running the tests
-----------------
If you want to perform further development of the nanopb core, or to verify
its functionality using your compiler and platform, you'll want to run the
test suite. The build rules for the test suite are implemented using Scons,
so you need to have that installed. To run the tests:
cd tests
scons
This will show the progress of various test cases. If the output does not
end in an error, the test cases were successful.
Note: Mac OS X by default aliases 'clang' as 'gcc', while not actually
supporting the same command line options as gcc does. To run tests on
Mac OS X, use: "scons CC=clang CXX=clang". Same way can be used to run
tests with different compilers on any platform.
all: index.html concepts.html reference.html security.html migration.html \
generator_flow.png
%.png: %.svg
rsvg $< $@
%.html: %.rst
rst2html --stylesheet=lsr.css --link-stylesheet $< $@
sed -i 's!</head>!<link href="favicon.ico" type="image/x-icon" rel="shortcut icon" />\n</head>!' $@
======================
Nanopb: Basic concepts
======================
.. include :: menu.rst
The things outlined here are the underlying concepts of the nanopb design.
.. contents::
Proto files
===========
All Protocol Buffers implementations use .proto files to describe the message
format. The point of these files is to be a portable interface description
language.
Compiling .proto files for nanopb
---------------------------------
Nanopb uses the Google's protoc compiler to parse the .proto file, and then a
python script to generate the C header and source code from it::
user@host:~$ protoc -omessage.pb message.proto
user@host:~$ python ../generator/nanopb_generator.py message.pb
Writing to message.h and message.c
user@host:~$
Modifying generator behaviour
-----------------------------
Using generator options, you can set maximum sizes for fields in order to
allocate them statically. The preferred way to do this is to create an .options
file with the same name as your .proto file::
# Foo.proto
message Foo {
required string name = 1;
}
::
# Foo.options
Foo.name max_size:16
For more information on this, see the `Proto file options`_ section in the
reference manual.
.. _`Proto file options`: reference.html#proto-file-options
Streams
=======
Nanopb uses streams for accessing the data in encoded format.
The stream abstraction is very lightweight, and consists of a structure (*pb_ostream_t* or *pb_istream_t*) which contains a pointer to a callback function.
There are a few generic rules for callback functions:
#) Return false on IO errors. The encoding or decoding process will abort immediately.
#) Use state to store your own data, such as a file descriptor.
#) *bytes_written* and *bytes_left* are updated by pb_write and pb_read.
#) Your callback may be used with substreams. In this case *bytes_left*, *bytes_written* and *max_size* have smaller values than the original stream. Don't use these values to calculate pointers.
#) Always read or write the full requested length of data. For example, POSIX *recv()* needs the *MSG_WAITALL* parameter to accomplish this.
Output streams
--------------
::
struct _pb_ostream_t
{
bool (*callback)(pb_ostream_t *stream, const uint8_t *buf, size_t count);
void *state;
size_t max_size;
size_t bytes_written;
};
The *callback* for output stream may be NULL, in which case the stream simply counts the number of bytes written. In this case, *max_size* is ignored.
Otherwise, if *bytes_written* + bytes_to_be_written is larger than *max_size*, pb_write returns false before doing anything else. If you don't want to limit the size of the stream, pass SIZE_MAX.
**Example 1:**
This is the way to get the size of the message without storing it anywhere::
Person myperson = ...;
pb_ostream_t sizestream = {0};
pb_encode(&sizestream, Person_fields, &myperson);
printf("Encoded size is %d\n", sizestream.bytes_written);
**Example 2:**
Writing to stdout::
bool callback(pb_ostream_t *stream, const uint8_t *buf, size_t count)
{
FILE *file = (FILE*) stream->state;
return fwrite(buf, 1, count, file) == count;
}
pb_ostream_t stdoutstream = {&callback, stdout, SIZE_MAX, 0};
Input streams
-------------
For input streams, there is one extra rule:
#) You don't need to know the length of the message in advance. After getting EOF error when reading, set bytes_left to 0 and return false. Pb_decode will detect this and if the EOF was in a proper position, it will return true.
Here is the structure::
struct _pb_istream_t
{
bool (*callback)(pb_istream_t *stream, uint8_t *buf, size_t count);
void *state;
size_t bytes_left;
};
The *callback* must always be a function pointer. *Bytes_left* is an upper limit on the number of bytes that will be read. You can use SIZE_MAX if your callback handles EOF as described above.
**Example:**
This function binds an input stream to stdin:
::
bool callback(pb_istream_t *stream, uint8_t *buf, size_t count)
{
FILE *file = (FILE*)stream->state;
bool status;
if (buf == NULL)
{
while (count-- && fgetc(file) != EOF);
return count == 0;
}
status = (fread(buf, 1, count, file) == count);
if (feof(file))
stream->bytes_left = 0;
return status;
}
pb_istream_t stdinstream = {&callback, stdin, SIZE_MAX};
Data types
==========
Most Protocol Buffers datatypes have directly corresponding C datatypes, such as int32 is int32_t, float is float and bool is bool. However, the variable-length datatypes are more complex:
1) Strings, bytes and repeated fields of any type map to callback functions by default.
2) If there is a special option *(nanopb).max_size* specified in the .proto file, string maps to null-terminated char array and bytes map to a structure containing a char array and a size field.
3) If there is a special option *(nanopb).max_count* specified on a repeated field, it maps to an array of whatever type is being repeated. Another field will be created for the actual number of entries stored.
=============================================================================== =======================
field in .proto autogenerated in .h
=============================================================================== =======================
required string name = 1; pb_callback_t name;
required string name = 1 [(nanopb).max_size = 40]; char name[40];
repeated string name = 1 [(nanopb).max_size = 40]; pb_callback_t name;
repeated string name = 1 [(nanopb).max_size = 40, (nanopb).max_count = 5]; | size_t name_count;
| char name[5][40];
required bytes data = 1 [(nanopb).max_size = 40]; | typedef struct {
| size_t size;
| uint8_t bytes[40];
| } Person_data_t;
| Person_data_t data;
=============================================================================== =======================
The maximum lengths are checked in runtime. If string/bytes/array exceeds the allocated length, *pb_decode* will return false.
Note: for the *bytes* datatype, the field length checking may not be exact.
The compiler may add some padding to the *pb_bytes_t* structure, and the nanopb runtime doesn't know how much of the structure size is padding. Therefore it uses the whole length of the structure for storing data, which is not very smart but shouldn't cause problems. In practise, this means that if you specify *(nanopb).max_size=5* on a *bytes* field, you may be able to store 6 bytes there. For the *string* field type, the length limit is exact.
Field callbacks
===============
When a field has dynamic length, nanopb cannot statically allocate storage for it. Instead, it allows you to handle the field in whatever way you want, using a callback function.
The `pb_callback_t`_ structure contains a function pointer and a *void* pointer called *arg* you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. A pointer to the *arg* is passed to the function, so that it can modify it and retrieve the value.
The actual behavior of the callback function is different in encoding and decoding modes. In encoding mode, the callback is called once and should write out everything, including field tags. In decoding mode, the callback is called repeatedly for every data item.
.. _`pb_callback_t`: reference.html#pb-callback-t
Encoding callbacks
------------------
::
bool (*encode)(pb_ostream_t *stream, const pb_field_t *field, void * const *arg);
When encoding, the callback should write out complete fields, including the wire type and field number tag. It can write as many or as few fields as it likes. For example, if you want to write out an array as *repeated* field, you should do it all in a single call.
Usually you can use `pb_encode_tag_for_field`_ to encode the wire type and tag number of the field. However, if you want to encode a repeated field as a packed array, you must call `pb_encode_tag`_ instead to specify a wire type of *PB_WT_STRING*.
If the callback is used in a submessage, it will be called multiple times during a single call to `pb_encode`_. In this case, it must produce the same amount of data every time. If the callback is directly in the main message, it is called only once.
.. _`pb_encode`: reference.html#pb-encode
.. _`pb_encode_tag_for_field`: reference.html#pb-encode-tag-for-field
.. _`pb_encode_tag`: reference.html#pb-encode-tag
This callback writes out a dynamically sized string::
bool write_string(pb_ostream_t *stream, const pb_field_t *field, void * const *arg)
{
char *str = get_string_from_somewhere();
if (!pb_encode_tag_for_field(stream, field))
return false;
return pb_encode_string(stream, (uint8_t*)str, strlen(str));
}
Decoding callbacks
------------------
::
bool (*decode)(pb_istream_t *stream, const pb_field_t *field, void **arg);
When decoding, the callback receives a length-limited substring that reads the contents of a single field. The field tag has already been read. For *string* and *bytes*, the length value has already been parsed, and is available at *stream->bytes_left*.
The callback will be called multiple times for repeated fields. For packed fields, you can either read multiple values until the stream ends, or leave it to `pb_decode`_ to call your function over and over until all values have been read.
.. _`pb_decode`: reference.html#pb-decode
This callback reads multiple integers and prints them::
bool read_ints(pb_istream_t *stream, const pb_field_t *field, void **arg)
{
while (stream->bytes_left)
{
uint64_t value;
if (!pb_decode_varint(stream, &value))
return false;
printf("%lld\n", value);
}
return true;
}
Field description array
=======================
For using the *pb_encode* and *pb_decode* functions, you need an array of pb_field_t constants describing the structure you wish to encode. This description is usually autogenerated from .proto file.
For example this submessage in the Person.proto file::
message Person {
message PhoneNumber {
required string number = 1 [(nanopb).max_size = 40];
optional PhoneType type = 2 [default = HOME];
}
}
generates this field description array for the structure *Person_PhoneNumber*::
const pb_field_t Person_PhoneNumber_fields[3] = {
PB_FIELD( 1, STRING , REQUIRED, STATIC, Person_PhoneNumber, number, number, 0),
PB_FIELD( 2, ENUM , OPTIONAL, STATIC, Person_PhoneNumber, type, number, &Person_PhoneNumber_type_default),
PB_LAST_FIELD
};
Extension fields
================
Protocol Buffers supports a concept of `extension fields`_, which are
additional fields to a message, but defined outside the actual message.
The definition can even be in a completely separate .proto file.
The base message is declared as extensible by keyword *extensions* in
the .proto file::
message MyMessage {
.. fields ..
extensions 100 to 199;
}
For each extensible message, *nanopb_generator.py* declares an additional
callback field called *extensions*. The field and associated datatype
*pb_extension_t* forms a linked list of handlers. When an unknown field is
encountered, the decoder calls each handler in turn until either one of them
handles the field, or the list is exhausted.
The actual extensions are declared using the *extend* keyword in the .proto,
and are in the global namespace::
extend MyMessage {
optional int32 myextension = 100;
}
For each extension, *nanopb_generator.py* creates a constant of type
*pb_extension_type_t*. To link together the base message and the extension,
you have to:
1. Allocate storage for your field, matching the datatype in the .proto.
For example, for a *int32* field, you need a *int32_t* variable to store
the value.
2. Create a *pb_extension_t* constant, with pointers to your variable and
to the generated *pb_extension_type_t*.
3. Set the *message.extensions* pointer to point to the *pb_extension_t*.
An example of this is available in *tests/test_encode_extensions.c* and
*tests/test_decode_extensions.c*.
.. _`extension fields`: https://developers.google.com/protocol-buffers/docs/proto#extensions
Message framing
===============
Protocol Buffers does not specify a method of framing the messages for transmission.
This is something that must be provided by the library user, as there is no one-size-fits-all
solution. Typical needs for a framing format are to:
1. Encode the message length.
2. Encode the message type.
3. Perform any synchronization and error checking that may be needed depending on application.
For example UDP packets already fullfill all the requirements, and TCP streams typically only
need a way to identify the message length and type. Lower level interfaces such as serial ports
may need a more robust frame format, such as HDLC (high-level data link control).
Nanopb provides a few helpers to facilitate implementing framing formats:
1. Functions *pb_encode_delimited* and *pb_decode_delimited* prefix the message data with a varint-encoded length.
2. Union messages and oneofs are supported in order to implement top-level container messages.
3. Message IDs can be specified using the *(nanopb_msgopt).msgid* option and can then be accessed from the header.
Return values and error handling
================================
Most functions in nanopb return bool: *true* means success, *false* means failure. There is also some support for error messages for debugging purposes: the error messages go in *stream->errmsg*.
The error messages help in guessing what is the underlying cause of the error. The most common error conditions are:
1) Running out of memory, i.e. stack overflow.
2) Invalid field descriptors (would usually mean a bug in the generator).
3) IO errors in your own stream callbacks.
4) Errors that happen in your callback functions.
5) Exceeding the max_size or bytes_left of a stream.
6) Exceeding the max_size of a string or array field
7) Invalid protocol buffers binary message.
This diff is collapsed.
=============================================
Nanopb: Protocol Buffers with small code size
=============================================
.. include :: menu.rst
Nanopb is an ANSI-C library for encoding and decoding messages in Google's `Protocol Buffers`__ format with minimal requirements for RAM and code space.
It is primarily suitable for 32-bit microcontrollers.
__ http://code.google.com/apis/protocolbuffers/
Overall structure
=================
For the runtime program, you always need *pb.h* for type declarations.
Depending on whether you want to encode, decode, or both, you also need *pb_encode.h/c* or *pb_decode.h/c*.
The high-level encoding and decoding functions take an array of *pb_field_t* structures, which describes the fields of a message structure. Usually you want these autogenerated from a *.proto* file. The tool script *nanopb_generator.py* accomplishes this.
.. image:: generator_flow.png
So a typical project might include these files:
1) Nanopb runtime library:
- pb.h
- pb_decode.h and pb_decode.c (needed for decoding messages)
- pb_encode.h and pb_encode.c (needed for encoding messages)
2) Protocol description (you can have many):
- person.proto (just an example)
- person.pb.c (autogenerated, contains initializers for const arrays)
- person.pb.h (autogenerated, contains type declarations)
Features and limitations
========================
**Features**
#) Pure C runtime
#) Small code size (2–10 kB depending on processor, plus any message definitions)
#) Small ram usage (typically ~300 bytes, plus any message structs)
#) Allows specifying maximum size for strings and arrays, so that they can be allocated statically.
#) No malloc needed: everything can be allocated statically or on the stack.
#) You can use either encoder or decoder alone to cut the code size in half.
#) Support for most protobuf features, including: all data types, nested submessages, default values, repeated and optional fields, packed arrays, extension fields.
#) Callback mechanism for handling messages larger than can fit in available RAM.
#) Extensive set of tests.
**Limitations**
#) Some speed has been sacrificed for code size.
#) Encoding is focused on writing to streams. For memory buffers only it could be made more efficient.
#) The deprecated Protocol Buffers feature called "groups" is not supported.
#) Fields in the generated structs are ordered by the tag number, instead of the natural ordering in .proto file.
#) Unknown fields are not preserved when decoding and re-encoding a message.
#) Reflection (runtime introspection) is not supported. E.g. you can't request a field by giving its name in a string.
#) Numeric arrays are always encoded as packed, even if not marked as packed in .proto. This causes incompatibility with decoders that do not support packed format.
#) Cyclic references between messages are supported only in callback mode.
Getting started
===============
For starters, consider this simple message::
message Example {
required int32 value = 1;
}
Save this in *message.proto* and compile it::
user@host:~$ protoc -omessage.pb message.proto
user@host:~$ python nanopb/generator/nanopb_generator.py message.pb
You should now have in *message.pb.h*::
typedef struct {
int32_t value;
} Example;
extern const pb_field_t Example_fields[2];
Now in your main program do this to encode a message::
Example mymessage = {42};
uint8_t buffer[10];
pb_ostream_t stream = pb_ostream_from_buffer(buffer, sizeof(buffer));
pb_encode(&stream, Example_fields, &mymessage);
After that, buffer will contain the encoded message.
The number of bytes in the message is stored in *stream.bytes_written*.
You can feed the message to *protoc --decode=Example message.proto* to verify its validity.
For a complete example of the simple case, see *example/simple.c*.
For a more complex example with network interface, see the *example/network_server* subdirectory.
Compiler requirements
=====================
Nanopb should compile with most ansi-C compatible compilers. It however
requires a few header files to be available:
#) *string.h*, with these functions: *strlen*, *memcpy*, *memset*
#) *stdint.h*, for definitions of *int32_t* etc.
#) *stddef.h*, for definition of *size_t*
#) *stdbool.h*, for definition of *bool*
If these header files do not come with your compiler, you can use the
file *extra/pb_syshdr.h* instead. It contains an example of how to provide
the dependencies. You may have to edit it a bit to suit your custom platform.
To use the pb_syshdr.h, define *PB_SYSTEM_HEADER* as *"pb_syshdr.h"* (including the quotes).
Similarly, you can provide a custom include file, which should provide all the dependencies
listed above.
Running the test cases
======================
Extensive unittests and test cases are included under the *tests* folder.
To build the tests, you will need the `scons`__ build system. The tests should
be runnable on most platforms. Windows and Linux builds are regularly tested.
__ http://www.scons.org/
In addition to the build system, you will also need a working Google Protocol
Buffers *protoc* compiler, and the Python bindings for Protocol Buffers. On
Debian-based systems, install the following packages: *protobuf-compiler*,
*python-protobuf* and *libprotobuf-dev*.
third_party/nanopb/docs/logo/logo.png

14.6 KiB

This diff is collapsed.
third_party/nanopb/docs/logo/logo16px.png

854 B

third_party/nanopb/docs/logo/logo48px.png

2.52 KiB

/*
Author: Peter Parente
Date: 2008/01/22
Version: 1.0 (modified)
Copyright: This stylesheet has been placed in the public domain - free to edit and use for all uses.
*/
body {
font: 100% sans-serif;
background: #ffffff;
color: black;
margin: 2em;
padding: 0em 2em;
}
p.topic-title {
font-weight: bold;
}
table.docinfo {
text-align: left;
margin: 2em 0em;
}
a[href] {
color: #436976;
background-color: transparent;
}
a.toc-backref {
text-decoration: none;
}
h1 a[href] {
color: #003a6b;
text-decoration: none;
background-color: transparent;
}
a.strong {
font-weight: bold;
}
img {
margin: 0;
border: 0;
}
p {
margin: 0.5em 0 1em 0;
line-height: 1.5em;
}
p a:visited {
color: purple;
background-color: transparent;
}
p a:active {
color: red;
background-color: transparent;
}
a:hover {
text-decoration: none;
}
p img {
border: 0;
margin: 0;
}
p.rubric {
font-weight: bold;
font-style: italic;
}
em {
font-style: normal;
font-family: monospace;
font-weight: bold;
}
pre {
border-left: 3px double #aaa;
padding: 5px 10px;
background-color: #f6f6f6;
}
h1.title {
color: #003a6b;
font-size: 180%;
margin-bottom: 0em;
}
h2.subtitle {
color: #003a6b;
border-bottom: 0px;
}
h1, h2, h3, h4, h5, h6 {
color: #555;
background-color: transparent;
margin: 0em;
padding-top: 0.5em;
}
h1 {
font-size: 150%;
margin-bottom: 0.5em;
border-bottom: 2px solid #aaa;
}
h2 {
font-size: 130%;
margin-bottom: 0.5em;
border-bottom: 1px solid #aaa;
}
h3 {
font-size: 120%;
margin-bottom: 0.5em;
}
h4 {
font-size: 110%;
font-weight: bold;
margin-bottom: 0.5em;
}
h5 {
font-size: 105%;
font-weight: bold;
margin-bottom: 0.5em;
}
h6 {
font-size: 100%;
font-weight: bold;
margin-bottom: 0.5em;
}
dt {
font-style: italic;
}
dd {
margin-bottom: 1.5em;
}
div.admonition, div.note, div.tip, div.caution, div.important {
margin: 2em 2em;
padding: 0em 1em;
border-top: 1px solid #aaa;
border-left: 1px solid #aaa;
border-bottom: 2px solid #555;
border-right: 2px solid #555;
}
div.important {
background: transparent url('../images/important.png') 10px 2px no-repeat;
}
div.caution {
background: transparent url('../images/caution.png') 10px 2px no-repeat;
}
div.note {
background: transparent url('../images/note.png') 10px 2px no-repeat;
}
div.tip {
background: transparent url('../images/tip.png') 10px 2px no-repeat;
}
div.admonition-example {
background: transparent url('../images/tip.png') 10px 2px no-repeat;
}
div.admonition-critical-example {
background: transparent url('../images/important.png') 10px 2px no-repeat;
}
p.admonition-title {
font-weight: bold;
border-bottom: 1px solid #aaa;
padding-left: 30px;
}
table.docutils {
text-align: left;
border: 1px solid gray;
border-collapse: collapse;
margin: 1.5em 0em;
}
table.docutils caption {
font-style: italic;
}
table.docutils td, table.docutils th {
padding: 0.25em 0.5em;
}
th.field-name {
text-align: right;
width: 15em;
}
table.docutils th {
font-family: monospace;
background-color: #f6f6f6;
vertical-align: middle;
}
table.field-list {
border: none;
}
div.sidebar {
margin: 2em 2em 2em 0em;
padding: 0em 1em;
border-top: 1px solid #aaa;
border-left: 1px solid #aaa;
border-bottom: 2px solid #555;
border-right: 2px solid #555;
}
p.sidebar-title {
margin-bottom: 0em;
color: #003a6b;
border-bottom: 1px solid #aaa;
font-weight: bold;
}
p.sidebar-subtitle {
margin-top: 0em;
font-style: italic;
color: #003a6b;
}
.. sidebar :: Documentation index
1) `Overview`_
2) `Concepts`_
3) `API reference`_
4) `Security model`_
5) `Migration from older versions`_
.. _`Overview`: index.html
.. _`Concepts`: concepts.html
.. _`API reference`: reference.html
.. _`Security model`: security.html
.. _`Migration from older versions`: migration.html
=====================================
Nanopb: Migration from older versions
=====================================
.. include :: menu.rst
This document details all the breaking changes that have been made to nanopb
since its initial release. For each change, the rationale and required
modifications of user applications are explained. Also any error indications
are included, in order to make it easier to find this document.
.. contents ::
Nanopb-0.3.2 (2015-01-24)
=========================
Add support for OneOfs
----------------------
**Rationale:** Previously nanopb did not support the *oneof* construct in
*.proto* files. Those fields were generated as regular *optional* fields.
**Changes:** OneOfs are now generated as C unions. Callback fields are not
supported inside oneof and generator gives an error.
**Required actions:** The generator option *no_unions* can be used to restore old
behaviour and to allow callbacks to be used. To use unions, one change is
needed: use *which_xxxx* field to detect which field is present, instead
of *has_xxxx*. Compare the value against *MyStruct_myfield_tag*.
**Error indications:** Generator error: "Callback fields inside of oneof are
not supported". Compiler error: "Message" has no member named "has_xxxx".
Nanopb-0.3.0 (2014-08-26)
=========================
Separate field iterator logic to pb_common.c
--------------------------------------------
**Rationale:** Originally, the field iteration logic was simple enough to be
duplicated in *pb_decode.c* and *pb_encode.c*. New field types have made the
logic more complex, which required the creation of a new file to contain the
common functionality.
**Changes:** There is a new file, *pb_common.c*, which must be included in
builds.
**Required actions:** Add *pb_common.c* to build rules. This file is always
required. Either *pb_decode.c* or *pb_encode.c* can still be left out if some
functionality is not needed.
**Error indications:** Linker error: undefined reference to
*pb_field_iter_begin*, *pb_field_iter_next* or similar.
Change data type of field counts to pb_size_t
---------------------------------------------
**Rationale:** Often nanopb is used with small arrays, such as 255 items or
less. Using a full *size_t* field to store the array count wastes memory if
there are many arrays. There already exists parameters *PB_FIELD_16BIT* and
*PB_FIELD_32BIT* which tell nanopb what is the maximum size of arrays in use.
**Changes:** Generator will now use *pb_size_t* for the array *_count* fields.
The size of the type will be controlled by the *PB_FIELD_16BIT* and
*PB_FIELD_32BIT* compilation time options.
**Required actions:** Regenerate all *.pb.h* files. In some cases casts to the
*pb_size_t* type may need to be added in the user code when accessing the
*_count* fields.
**Error indications:** Incorrect data at runtime, crashes. But note that other
changes in the same version already require regenerating the files and have
better indications of errors, so this is only an issue for development
versions.
Renamed some macros and identifiers
-----------------------------------
**Rationale:** Some names in nanopb core were badly chosen and conflicted with
ISO C99 reserved names or lacked a prefix. While they haven't caused trouble
so far, it is reasonable to switch to non-conflicting names as these are rarely
used from user code.
**Changes:** The following identifier names have changed:
* Macros:
* STATIC_ASSERT(x) -> PB_STATIC_ASSERT(x)
* UNUSED(x) -> PB_UNUSED(x)
* Include guards:
* _PB_filename_ -> PB_filename_INCLUDED
* Structure forward declaration tags:
* _pb_field_t -> pb_field_s
* _pb_bytes_array_t -> pb_bytes_array_s
* _pb_callback_t -> pb_callback_s
* _pb_extension_type_t -> pb_extension_type_s
* _pb_extension_t -> pb_extension_s
* _pb_istream_t -> pb_istream_s
* _pb_ostream_t -> pb_ostream_s
**Required actions:** Regenerate all *.pb.c* files. If you use any of the above
identifiers in your application code, perform search-replace to the new name.
**Error indications:** Compiler errors on lines with the macro/type names.
Nanopb-0.2.9 (2014-08-09)
=========================
Change semantics of generator -e option
---------------------------------------
**Rationale:** Some compilers do not accept filenames with two dots (like
in default extension .pb.c). The *-e* option to the generator allowed changing
the extension, but not skipping the extra dot.
**Changes:** The *-e* option in generator will no longer add the prepending
dot. The default value has been adjusted accordingly to *.pb.c* to keep the
default behaviour the same as before.
**Required actions:** Only if using the generator -e option. Add dot before
the parameter value on the command line.
**Error indications:** File not found when trying to compile generated files.
Nanopb-0.2.7 (2014-04-07)
=========================
Changed pointer-type bytes field datatype
-----------------------------------------
**Rationale:** In the initial pointer encoding support since nanopb-0.2.5,
the bytes type used a separate *pb_bytes_ptr_t* type to represent *bytes*
fields. This made it easy to encode data from a separate, user-allocated
buffer. However, it made the internal logic more complex and was inconsistent
with the other types.
**Changes:** Dynamically allocated bytes fields now have the *pb_bytes_array_t*
type, just like statically allocated ones.
**Required actions:** Only if using pointer-type fields with the bytes datatype.
Change any access to *msg->field.size* to *msg->field->size*. Change any
allocation to reserve space of amount *PB_BYTES_ARRAY_T_ALLOCSIZE(n)*. If the
data pointer was begin assigned from external source, implement the field using
a callback function instead.
**Error indications:** Compiler error: unknown type name *pb_bytes_ptr_t*.
Nanopb-0.2.4 (2013-11-07)
=========================
Remove the NANOPB_INTERNALS compilation option
----------------------------------------------
**Rationale:** Having the option in the headers required the functions to
be non-static, even if the option is not used. This caused errors on some
static analysis tools.
**Changes:** The *#ifdef* and associated functions were removed from the
header.
**Required actions:** Only if the *NANOPB_INTERNALS* option was previously
used. Actions are as listed under nanopb-0.1.3 and nanopb-0.1.6.
**Error indications:** Compiler warning: implicit declaration of function
*pb_dec_string*, *pb_enc_string*, or similar.
Nanopb-0.2.1 (2013-04-14)
=========================
Callback function signature
---------------------------
**Rationale:** Previously the auxilary data to field callbacks was passed
as *void\**. This allowed passing of any data, but made it unnecessarily
complex to return a pointer from callback.
**Changes:** The callback function parameter was changed to *void\*\**.
**Required actions:** You can continue using the old callback style by
defining *PB_OLD_CALLBACK_STYLE*. Recommended action is to:
* Change the callback signatures to contain *void\*\** for decoders and
*void \* const \** for encoders.
* Change the callback function body to use *\*arg* instead of *arg*.
**Error indications:** Compiler warning: assignment from incompatible
pointer type, when initializing *funcs.encode* or *funcs.decode*.
Nanopb-0.2.0 (2013-03-02)
=========================
Reformatted generated .pb.c file using macros
---------------------------------------------
**Rationale:** Previously the generator made a list of C *pb_field_t*
initializers in the .pb.c file. This led to a need to regenerate all .pb.c
files after even small changes to the *pb_field_t* definition.
**Changes:** Macros were added to pb.h which allow for cleaner definition
of the .pb.c contents. By changing the macro definitions, changes to the
field structure are possible without breaking compatibility with old .pb.c
files.
**Required actions:** Regenerate all .pb.c files from the .proto sources.
**Error indications:** Compiler warning: implicit declaration of function
*pb_delta_end*.
Changed pb_type_t definitions
-----------------------------
**Rationale:** The *pb_type_t* was previously an enumeration type. This
caused warnings on some compilers when using bitwise operations to set flags
inside the values.
**Changes:** The *pb_type_t* was changed to *typedef uint8_t*. The values
were changed to *#define*. Some value names were changed for consistency.
**Required actions:** Only if you directly access the `pb_field_t` contents
in your own code, something which is not usually done. Needed changes:
* Change *PB_HTYPE_ARRAY* to *PB_HTYPE_REPEATED*.
* Change *PB_HTYPE_CALLBACK* to *PB_ATYPE()* and *PB_ATYPE_CALLBACK*.
**Error indications:** Compiler error: *PB_HTYPE_ARRAY* or *PB_HTYPE_CALLBACK*
undeclared.
Nanopb-0.1.6 (2012-09-02)
=========================
Refactored field decoder interface
----------------------------------
**Rationale:** Similarly to field encoders in nanopb-0.1.3.
**Changes:** New functions with names *pb_decode_\** were added.
**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
the old functions. Recommended action is to replace any calls with the newer
*pb_decode_\** equivalents.
**Error indications:** Compiler warning: implicit declaration of function
*pb_dec_string*, *pb_dec_varint*, *pb_dec_submessage* or similar.
Nanopb-0.1.3 (2012-06-12)
=========================
Refactored field encoder interface
----------------------------------
**Rationale:** The old *pb_enc_\** functions were designed mostly for the
internal use by the core. Because they are internally accessed through
function pointers, their signatures had to be common. This led to a confusing
interface for external users.
**Changes:** New functions with names *pb_encode_\** were added. These have
easier to use interfaces. The old functions are now only thin wrappers for
the new interface.
**Required actions:** By defining NANOPB_INTERNALS, you can still keep using
the old functions. Recommended action is to replace any calls with the newer
*pb_encode_\** equivalents.
**Error indications:** Compiler warning: implicit declaration of function
*pb_enc_string*, *pb_enc_varint, *pb_enc_submessage* or similar.
This diff is collapsed.
======================
Nanopb: Security model
======================
.. include :: menu.rst
.. contents ::
Importance of security in a Protocol Buffers library
====================================================
In the context of protocol buffers, security comes into play when decoding
untrusted data. Naturally, if the attacker can modify the contents of a
protocol buffers message, he can feed the application any values possible.
Therefore the application itself must be prepared to receive untrusted values.
Where nanopb plays a part is preventing the attacker from running arbitrary
code on the target system. Mostly this means that there must not be any
possibility to cause buffer overruns, memory corruption or invalid pointers
by the means of crafting a malicious message.
Division of trusted and untrusted data
======================================
The following data is regarded as **trusted**. It must be under the control of
the application writer. Malicious data in these structures could cause
security issues, such as execution of arbitrary code:
1. Callback, pointer and extension fields in message structures given to
pb_encode() and pb_decode(). These fields are memory pointers, and are
generated depending on the message definition in the .proto file.
2. The automatically generated field definitions, i.e. *pb_field_t* lists.
3. Contents of the *pb_istream_t* and *pb_ostream_t* structures (this does not
mean the contents of the stream itself, just the stream definition).
The following data is regarded as **untrusted**. Invalid/malicious data in
these will cause "garbage in, garbage out" behaviour. It will not cause
buffer overflows, information disclosure or other security problems:
1. All data read from *pb_istream_t*.
2. All fields in message structures, except callbacks, pointers and extensions.
(Beginning with nanopb-0.2.4, in earlier versions the field sizes are partially unchecked.)
Invariants
==========
The following invariants are maintained during operation, even if the
untrusted data has been maliciously crafted:
1. Nanopb will never read more than *bytes_left* bytes from *pb_istream_t*.
2. Nanopb will never write more than *max_size* bytes to *pb_ostream_t*.
3. Nanopb will never access memory out of bounds of the message structure.
4. After pb_decode() returns successfully, the message structure will be
internally consistent:
- The *count* fields of arrays will not exceed the array size.
- The *size* field of bytes will not exceed the allocated size.
- All string fields will have null terminator.
5. After pb_encode() returns successfully, the resulting message is a valid
protocol buffers message. (Except if user-defined callbacks write incorrect
data.)
Further considerations
======================
Even if the nanopb library is free of any security issues, there are still
several possible attack vectors that the application author must consider.
The following list is not comprehensive:
1. Stack usage may depend on the contents of the message. The message
definition places an upper bound on how much stack will be used. Tests
should be run with all fields present, to record the maximum possible
stack usage.
2. Callbacks can do anything. The code for the callbacks must be carefully
checked if they are used with untrusted data.
3. If using stream input, a maximum size should be set in *pb_istream_t* to
stop a denial of service attack from using an infinite message.
4. If using network sockets as streams, a timeout should be set to stop
denial of service attacks.
5. If using *malloc()* support, some method of limiting memory use should be
employed. This can be done by defining custom *pb_realloc()* function.
Nanopb will properly detect and handle failed memory allocations.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please to comment