CHANGES
Pike 8.1: Changes since Pike 8.0 (scratch area for future release notes)
------------------------------------------------------------------------
New language features
---------------------
o Unlimited character constant size.
There is no longer any limit to the size of character constants,
e.g. 'acdefghijk' creates the bignum 0x61636465666768696a6b.
o 'auto' type added.
This is much like a strict typed mixed. The actual type is deduced
compile-time.
Typical use is as the type for variables in foreach when looping over
complexly typed values, or as the type in macro-defined functions and
similar.
auto can be used as return type for a function, it will be the
type-union of all the types of the return statements in the
function.
o typeof(X) can now be used as a type.
The main use case is in macros, but it might be useful for other
things as well (as an example in a typedef).
typedef typeof(Val.true)|typeof(Val.false) bool;
typeof, when used like this, behaves very much like the C++
decltype() expression.
o ** operator added. It's exponentiation and works with most
combination of numerical types (int,float,Gmp.mpq,Gmp.mpf,Gmp.mpz)
`** and ``** operator overloading functions added.
This introduces one incompatible change: Previously the "pow"
function called a "_pow" function in the first argument if it was an
object. It has now been changed to also use `** (or, rather, pow()
is now implemented using predef::`**()).
o Three pass compiler.
An extra pass of the compiler has been added. It is used to properly
resolve types in forward referring expressions, and is only run when
needed.
o #pragma disassemble
This directive is now available even in pikes compiled --without-debug.
The byte code output is now interleaved with the generated machine code
on selected architectures.
o Complain about redundant backslash escapes.
o '__weak__' modifier added.
It is now possible to declare object variables to have weak references.
o Support for arguments to implicit lambdas added.
Implicit lambdas now behave as if they were declared as
lambda(mixed ... __ARGS__).
Support for this feature can be detected with #ifdef __ARGS__.
o Function local function declarations are now statements
(and not expressions). This means that there is no longer
any need to terminate them with a semicolon.
o Anonymous class definitions are now always expressions.
o Function local class definitions are now statements
(and not expressions). This means that there is no longer
any need to terminate them with a semicolon.
o Complain about shadowed variant functions.
o Machine code support for more architectures.
There's now machine code support for arm32 and arm64.
o Fixed multiple integer over- and underflow bugs.
o Extended sscanf %O
sscanf() is now able to parse all base types that sprintf %O can
output. It can now also parse most literal constants pike accepts,
including integers prefixed with 0x and 0b, and floating point
numbers in scientific notation.
o Returning void
It is now possible to return void responses from void functions.
void a() { return; }
void b() { return a(); }
o Protocols.DNS now supports encoding and decoding CAA RRs.
Bug fixes
---------
o Operator functions
Calling operator functions with more than two arguments will now
work correctly for objects, where previously only the first two
objects where added.
When adding arrays, multisets and mappings, UNDEFINED will now
always be ignored. Previously it was only ignored when first in the
argument list, otherwise an exception would be thrown.
The LFUNs will now be called with a consistent number of
arguments. Pike implementations rarely implemented varargs on
operator LFUNs, so this change should address many potential hidden
errors.
o Protocols.HTTP.Server
The server module will now read payloads for HTTP PUT requests, just
as any other method. Previously it would stop reading the body and
it was up to the caller to read enough data from the socket and
combine with already read data.
Setting "connection" header in the "extra_heads" to Request object
method response_and_finish() will now control if keep-alive should be
used or not. Otherwise it will be decided by the clients request
headers, as previously.
The headers "content-type", "content-range", "content-length",
"server", "date" and "last-modified" will not be added or
overwritten if provided in the "extra_heads".
Header names in "extra_heads" will not have their case modified.
Incompatible changes
--------------------
o Sql.Sql is no longer a wrapper class, but a function.
The wrapper class has been obsoleted by introduction of the
new base class __builtin.Sql.Connection. Note that it is
still possible to use Sql.Sql as the type of connection objects.
o Gz.crc32 and Nettle.crc32c now only return positive results.
o glob() has changed.
The main incompatibilities are that [ and \ are now special
characters in the pattern, and if you check the return value against
1 instead of checking if it's true, and use an array as the first
argument you will have to change your code.
This is in order to extend the glob function to cover what
'standard' glob functions do:
glob() now accepts quotes (\* to match a single *, as an example)
and handles ranges ([abc] for a, b or c, [a-z0-9] for a single
character between a and z or 0-9
You can also negate a range using ^ or ! ([^a-zA-Z] as an example).
When the first argument (the pattern) to glob is an array, glob now
returns which pattern in the array matched.
o hash() has been changed to use siphash. The old hash function is
available under the name hash_8_0(). Note that this is not a
cryptographic hash function.
o Stdio.UDP()->send() no longer throws errors on EMSGSIZE and EWOULDBLOCK.
o If a thread exits with by throwing, the thrown value is propagated to wait().
o String.trim_all_whites() renamed String.trim().
o Floats are no longer by default sorted before added. This may reduce
the precision but increases the speed of adding large number of
floats by a factor of 500. Applications handling floats with large
differences in magnitude need to apply the appropriate sorting
before arithmetics. As `+ was the only operator that performed
sorting, and functions like Array.sum did not, this was already a
concern.
o Returning UNDEFINED from `+ and ``+ is not allowed and will cause an
exception.
o RegGetValue(), RegGetKeyNames(), RegGetKeyValues(), openlog(),
syslog() and closelog() have been moved to System.
New modules
-----------
o ADT.Scheduler & ADT.TreeScheduler
These are variants of ADT.Heap where elements typically aren't
removed from the heap.
o Apple.Keychain
Parser for Apple Keychain format files (like the ones in
/System/Library/Keychains/).
o __builtin.Sql
Generic base classes for Sql API modules.
o Filesystem.Zip
o Function.bind()
Partially evaluate a function. Similar to Function.curry, but with
better granularity.
o Standards.HPack (RFC 7541)
o Stdio.FakePipe
A simulated Stdio.Pipe.
o Parser.Markdown
o Crypto.Checksum
This module collect non-cryptographic checksums. Support for crc32,
adler32 and Castagnoli CRC (CRC32C).
NB: In the future these may be amended to support the Crypto.Hash API.
o Parser.ECMAScript
This module simply provides a token splitter for
ECMAScript/JavaScript.
o Web.EngineIO & Web.SocketIO
o Protocols.HTTP2
o Bittorrent.DHT
o Standards.MsgPack
o Web.Auth & Web.Api
New features
------------
o predef::equal()
equal() on functions now checks if their definitions are same identifier
in the same program.
o predef::gc()
gc() called with a weak mapping as argument now removes weak references
that are only held by that mapping.
o ADT.Heap
- An indirection object ADT.Heap.Element has been added to make it
possible to optimize several heap operations.
- Added low_pop().
o Crypto & Nettle
- Added Curve25519 and EdDSA25519.
o Filesystem.Monitor
The filesystem monitoring system now uses accelleration via
Inotify et al.
o Gmp
- mpf is now implemented using gmpf if the library is available.
- Improved support for GMP 5.0 and later.
o GTK2
Multiple runtime fixes.
o JOSE (JSON Object Signing and Encryption)
Some low-level API support has been added to the Crypto and Web
modules to support parts of RFC 7515 - 7520.
o MasterObject
- Protect against the same file being compiled concurrently
in multiple threads.
- cast_to_program() and cast_to_object() should now be thread safe.
o CompilerEnvironment()->lock()
Access to the compiler lock.
o Crypto.ECC.Curve.Point
A point on an ECC curve.
o Parser.Pike
Support new language features.
o Protocols.DNS.server
Derived classes can now override report_decode_error() and
handle_decode_error() to change how errors while decoding a DNS
packet are reported and handled.
o Protocols.WebSocket
Multiple API changes.
o Random rewrite
The random functions have been rewritten to ensure security by
default. random_string() and random() now get their data directly
from the operating system random generator, i.e. /dev/urandom on
most unixen. This is about half the speed compared with the
random_string function in Pike 8.0, but is now as secure as the
system random generator.
For consumers of random data that have additional requirements,
different random generators are exposed in the new module
Random. The module has the following generators:
- Random.Interface
This is not actually a generator, but an interface class that is
inherited into all the other generators. It contains code that can
turn the output from the random_string method into random numbers
with different limits without introducing bias. It also contains
code for all the different variants of random() on different
types. This is currently not possible to implement in Pike code,
as the typing is too complicated and it is not possible to access
private/protected _random methods in objects.
- Random.System
This generator maps directly on top of the system random
generator. This is the default generator used for random() and
random_string().
- Random.Deterministic
This generator creates the same sequence of random numbers for a
given seed, with good pseudo random properties.
- Random.Hardware
This generator accesses the hardware random generator, when
available.
- Random.Fast
This generator takes entropy from the Random.System, but feeds
it into a cryptographic pseudo random number generator to be
able to output data fast. This is not the default random number
generator to avoid loading crypto code on every startup.
Comparing the different generators with each other gives the
following approximate speeds on a Linux system with hardware
random support:
Random.System 1.0
Pike 8.0 random_string 0.45
Random.Hardware 0.25
Random.Fast 0.20
Random.Deterministic 0.20
Objects implementing the _random lfun now get two arguments, the
current random_string() and random() functions. This is convenient
for C-level functions that doesn't have to look up functions
themselves. Note that it is possible for a user to replace these
with non-conforming functions (returning values of the wrong type,
strings of the wrong length or shift size, and values outside the
given range) or even non-functions.
All code in Pike that uses random now uses the current random
functions (though in some cases fixed at object creation). This
allows for repeatable results if the random functions are replaced
with a deterministic random generator, such as
Random.Deterministic. Example:
Random.Deterministic rnd = Random.Deterministic( seed );
add_constant( "random_string", rnd->random_string );
add_constant( "random", rnd->random );
o Sql
- Most Sql C-modules converted to cmod.
- Added next_result().
- ODBC & tds: Support more datatypes.
- ODBC: Support big_typed_query().
- pgsql: Lots of changes and fixes.
o SSL
- Support session tickets.
- Support Negotiated FF-DHE.
- Support client certificates.
- Support ALPN.
- Prefer AEAD suites to CBC suites.
- SSL.File supports set_buffer_mode().
o Standards.PKCS
Support PKCS#8 private keys.
o String.Buffer & Stdio.Buffer
Added _search().
o The self testing framework now supports *.test-files.
o Thread
- _sprintf() improvements: Thread.Mutex now prints the ID of the thread
holding the lock, and thread IDs are shown as hexadecimal numbers.
- Thread.Farm now supports a callback for thread creation and termination
for the purpose of tracking thread names.
o sprintf %x
%X and %x can now be used on 8-bit wide strings to get a hexadecimal
representation of their contents. Just calling sprintf("%x",data) is
the same as calling String.string2hex(data).
o Unicode 10.0.0.
o Unicode.is_whitespace()
This new function returns if a unicode character is a whitespace
characer or not.
Deprecated symbols and modules
------------------------------
o Sql.mysql_result and Sql.mysqls_result have been deprecated.
Use Sql.Result instead.
o call_function() has been deprecated. Use `()() instead.
Removed features and modules
----------------------------
o Compatibility for Pike versions before 7.8 is no longer available.
o GTK1 library is deprecated, so glue code is removed.
C-level API changes
-------------------
o The contract for functions is now changed so that a function is no
longer required to clean the stack. The topmost value of the stack
will be regarded as the return value and the rest of the items on
the stack, compared to before the function arguments were pushed,
will be popped and discarded. Efuns still have to clean their stack
as previously.
o Removed add_function, pike_add_function, pike_add_function2,
simple_add_variable, map_variable and MAP_VARIABLE. This removes the
remaining few API:s where text types were used. Use ADD_FUNCTION and
PIKE_MAP_VARIABLE instead.
o Removed the functions my_rand and my_srand. Use the random functions
on the stack for _random lfuns, the push_random_string or look up
the random function from get_builtin_constants(). For deterministic
pseudo random, create a private Random.Deterministic object.
o The preprocessor has been converted into a cmod, and been modified
to use more standard Pike datatypes.
o The preprocessor-specific hashtable implementation has been removed.
o The gdb_backtraces() function is now available also --without-debug.
o There's now support to block mapping hashtables from being shrunk
on map_delete().
o guaranteed_memset() is replaced with secure_zero() which fills a
buffer with zero. On x86 SSE2 is used to zero the memory without
loading it into the CPU cache, as this function is typically used
before calling free() on memory with cryptographic key data.
Documentation
-------------
o RFC references added.
o Character encoding issues fixed.
o Added @enum/@endenum markup.
o Support undocumented enums with documented constants.
Building and installing
-----------------------
o GMP 4.1 or later is now required.
o C99 assumed
The configure tests will not check for functions defined in C99
anymore and C99 compiler support is assumed.
Optimizations
-------------
o Removed the GC marker hash table. For types which require GC markers,
they are now allocated as parf of the data type. This significantly
improves GC performance ( up to a factor if 2 in some situations ).