Added SILC Thread Pool API.

[silc.git] / TODO

TODO for 1.2 And Beyond
=======================

NOTE: Any item that doesn't have (***DONE) in it, isn't done yet.  The
(***TESTING NEEDED) means that the item has been done but not yet properly
tested.

NOTE: A TODO entry does not mean that it is ever going to be done.  Some
of the entries may be just ideas, good, bad or ugly.  If you want to work
on some of the TODO entries simply let us know about it by dropping a note
to silc-devel mailing list or appear on 'silc' channel on SILCNet.


General
=======

 o Create apps/tutorial containing various Toolkit API tutorials.

 o The Toolkit split.  The Toolkit is to be splitted in parts.  How many
   parts and what the parts are isn't decided yet.  Each part is a separate
   software package.  Current thinking is of the following:

   SILC Toolkit                 SILC protocol, client and server library
   SILC Runtime Toolkit         runtime library
   SILC Crypto Toolkit          crypto, asn1, math, skr, pgp, etc.

   The rationale for this is of course that other than SILC projects
   might like to use the various libraries SILC Toolkit provides, but
   naturally they don't want the bloat of SILC protocol related stuff.

   The Runtime library in SILC Toolkit is a general purpose runtime library,
   like Glib and APR are.  The runtime library is to be developed further
   to provide alternative to Glib and APR.

   The Crypto library in SILC Toolkit is a general purpose crypto library
   providing pretty nice APIs compared to many other crypto libraries,
   especially OpenSSL.  The Crypto library is to be developed further
   to include support for OpenPGP, X.509 and SSH2.


lib/silccore
============

 o SILC_PACKET_FLAG_ACK support.  Implement ACK packet and packet payload
   to silcpacket.c.

 o All payload encoding routines should take SilcStack as argument.

 o Remove SilcCommandCb from silccommand.h.

 o All payload test routines into lib/silccore/tests/.


lib/silcclient, The Client Library
==================================

 o Giving WHOIS for nick that doesn't exist should remove any same
   named entries from the client cache.

 o peer-to-peer private messages

 o Private message key request notification to application.  See XXX in
   client_prvmsg.c.

 o in JOIN notify handle resolving that timedout.  Currently the user is
   never joined the channel if this happens.  What to do if message is
   received from user that hasn't been resolved/joined?

 o Message ACKing support.

 o in /cmode and /cumode with +r, maybe the public key and private key
   could be just some "string", which would then match to "string.pub" and
   "string.prv".

 o If the SILC Events (see below) are implemented, perhaps client library
   should provide events so that application developer has a choice of
   developing the SILC app with callbacks or with events.


Runtime library, lib/silcutil/
==============================

 o Fix universal time decoding (doesn't accept all formats) in silctime.c.

 o Add functions to manipulate environment variables.

   SilcBool silc_setenv(const char *variable, const char *value);
   const char *silc_getenv(const char *variable);
   SilcBool silc_clearenv(const char *variable);

 o Add functions to loading shared/dynamic object symbols (replaces the
   SIM library (lib/silcsim) and introduces generic library).  Add this
   to lib/silcutil/silcdll.[ch].

   SilcDll silc_dll_load(const char *object_path, SilcDllFlags flags);
   void silc_dll_close(SilcDll dll);
   void *silc_dll_getsym(SilcDll dll, const char *symbol);
   const char *silc_dll_error(SilcDll dll);

 o Add directory opening/traversing functions

 o silc_getopt routines

 o silc_hash_table_replace -> silc_hash_table_set.  Retain support for
   silc_hash_table_replace as macro.

 o The SILC Event signals.  Asynchronous events that can be created,
   connected to and signalled.  Either own event routines or glued into
   SilcSchedule:

   SilcTask silc_schedule_task_add_event(SilcSchedule schedule,
                                         const char *event, ...);
   SilcBool silc_schedule_event_connect(SilcSchedule schedule,
                                        const char *event,
                                        SilcTaskCallback event_callback,
                                        void *context);
   SilcBool silc_schedule_event_signal(SilcSchedule schedule,
                                       const char *event, ...);

   Example:
     silc_schedule_task_add_event(schedule, "connected",
                                  SILC_PARAM_UI32_INT,
                                  SILC_PARAM_BUFFER,
                                  SILC_PARAM_END);
     silc_schedule_event_connect(schedule, "connected", connected_cb, ctx);
     silc_schedule_event_signal(schedule, "connected", integer, buf,
                                 SILC_PARAM_END);
     SILC_TASK_CALLBACK(connected_cb)
     {
       FooCtx ctx = context;
       va_list args;
       SilcUInt32 integer;
       SilcBuffer buf;

       va_start(args, context);
       integer = va_arg(args, SilcUInt32);
       buf = va_arg(args, SilcBuffer);
       va_end(args);
       ...
     }

   Problems: Events would be SilcSchedule specific, and would not work on
   multi-thread/multi-scheduler system.  The events should be copyable
   between schedulers.  Another problem is the signal delivery.  Do we
   deliver them synchronously possibly from any thread to any other thread
   or do we deliver them through the target schedulers.  If we use the
   schedulers then signalling would be asynchronous (data must be
   duplicated and later freed) which is not very nice.

 o If the event signals are added, the SILC_PARAM_* stuff needs to be
   moved from silcbuffmt.h to silctypes.h or something similar.

 o In case the SILC Events are done we shall create a new concept of
   parent and child SilcSchedule's.  When new SilcSchedule is created a
   parent can be associated to it.  This association could be done either
   directly by the parent or by any other children.  This way the signals
   would in effect be global and would reach all children schedulers.

   This relationship would be associative only.  The schedulers are still
   independent and run independently from each other.   All schedulers
   would be linked and could be accessed from any of the schedulers.
   It should be possible to retrieve the parent and enumate all children
   from any of the schedulers.

   SilcSchedule silc_schedule_init(int max_tasks, void *app_context,
                                   SilcSchedule parent);
   SilcSchedule silc_schedule_get_parent(SilcSchedule schedule);

 o Additional scheduler changes: optimize silc_schedule_wakeup.  Wakeup
   only if the scheduler is actually waiting something.  If it is
   delivering tasks wakeup is not needed.

 o Structured log messages to Log API.  Allows machine readable log
   messages.  Would allow sending of any kind of data in a log message.

 o Base64 to an own API

 o Timer API

 o Add builtin SOCKS and HTTP Proxy support, well the SOCKS at least.
   SILC currently supports SOCKS4 and SOCKS5 but it needs to be compiled
   in separately.

 o silc_stringprep to non-allocating version.

 o SilcStack aware SilcHashTable.

 o SilcStack aware SilcDList.

 o Thread pool API.  Add this to lib/silcutil/silcthread.[ch].  (***DONE)

 o Fast mutex implementation.  Fast rwlock implementation.  Mutex and
   rwlock implementation using atomic operations.

 o Compression routines are missing.  The protocol supports packet
   compression thus it must be implemented.  SILC Zip API must be
   defined.

 o Add new functions to SilcStack API in lib/silcutil/silcstack.[ch].  Add
   silc_stack_[set|get]_alignment.  It defines the default alignment used
   when allocating memory from stack.  It can be used to specify special
   alignments too when needed (such as for hardware devices like crypto
   accelerators).  Move also the low level silc_stack_malloc and
   silc_stack_realloc from silcstack_i.h to silcstack.h.  Remove the
   _ua unaligned memory allocation routines.  Remove unaligned memory
   allocation possibility.

 o Add '%@' format to silc_snprintf functions.  It marks for external
   rendering function of following type:

     /* Snprintf rendering function.  The `data' is rendered into a string
        and allocated string is returned.  If NULL is returned the
        rendering is skipped and ignored.  If the returned string does
        not fit to the destination buffer it may be truncated. */
     typedef char *(*SilcSnprintfRender)(void *data);

   It can work like following:

   char *id_renderer(void *data)
   {
     char tmp[32];
     id_to_str(tmp, sizeof(tmp), (SilcID *)data);
     return strdup(tmp);
   }

   silc_snprintf(buf, sizeof(buf), "Client ID %@", id_renderer, client_id);

 (o Generic SilcStatus or SilcResult that includes all possible status and
    error conditions, including those of SILC protocol.  Though, the SILC
    protocol related status (currently in silcstatus.h) cannot be in
    runtime library) maybe

 (o SILC specific socket creation/closing routines to silcnet.h, wrappers
  to all send(), recv(), sendto() etc.  Bad thing is that we'd have to
  define all socket options, sockaddrs, etc.) maybe

 (o mmap) maybe


lib/silcutil/symbian/
=====================

 o Something needs to be thought to the logging globals as well,
   like silc_debug etc.  They won't work on EPOC.  Perhaps logging
   and debugging is to be disabled on EPOC.  The logging currently works
   by it cannot be controlled, same with debugging.


SFTP Library, lib/silcsftp/
===========================

 o Read prefetch (read-ahead, reading ahead of time).  Maybe if this can
   be done easily.


SKR Library, lib/silcskr/
=========================

 o Add fingerprint as search constraint.

 o Add OpenPGP support.  Adding, removing, fetching PGP keys.  (Keyring
   support?)

 o Add support for importing public keys from a directory and/or from a
   file.  Add support for exporting the repository (different formats for
   different key types?).

 o Change the entire silc_skr_find API.  Remove SilcSKRFind and just simply
   add the find constraints as variable argument list to silc_skr_find, eg:

  silc_skr_find(skr, schedule, callback, context,
                SILC_SKR_FIND_PUBLIC_KEY, public_key,
                SILC_SKR_FIND_COUNTRY, "FI",
                SILC_SKR_FIND_USAGE, SILC_SKR_USAGE_AUTH,
                SILC_SKR_FIND_END);

   NULL argument would be ignored and skipped.

 o Add OR logical rule in addition of the current default AND, eg:

  // Found key(s) MUST have this public key AND this country.
  silc_skr_find(skr, schedule, callback, context,
                SILC_SKR_FIND_RULE_AND,
                SILC_SKR_FIND_PUBLIC_KEY, public_key,
                SILC_SKR_FIND_COUNTRY, "FI",
                SILC_SKR_FIND_END);

  // Found key(s) MUST have this public key OR this key context
  silc_skr_find(skr, schedule, callback, context,
                SILC_SKR_FIND_RULE_OR,
                SILC_SKR_FIND_PUBLIC_KEY, public_key,
                SILC_SKR_FIND_CONTEXT, key_context,
                SILC_SKR_FIND_END);

 o SilcStack to SKR API.


Crypto Library, lib/silccrypt/
==============================

 o SilcStack to APIs.

 o Add fingerprint to SilcSILCPublicKey and retrieval to silcpk.h, and
   possibly to silcpkcs.h.

   /* Return fingerprint of the `public_key'.  Returns also the algorithm
      that has been used to make the fingerprint. */
   const unsigned char *
   silc_pkcs_get_fingerprint(SilcPublicKey public_key,
                             const char **hash_algorithm,
                             SilcUInt32 *fingerprint_len);

 o Change SILC PKCS API to asynchronous, so that accelerators can be used.
   All PKCS routines should now take callbacks as argument and they should
   be delivered to SilcPKCSObject and SilcPKCSAlgorithm too.

   /* Signature computation callback */
   typedef void (*SilcPKCSSignCb)(SilcBool success,
                                  const unsigned char *signature,
                                  SilcUInt32 signature_len,
                                  void *context);

   /* Signature verification callback */
   typedef void (*SilcPKCSVerifyCb)(SilcBool success, void *context);

   /* Encryption callback */
   typedef void (*SilcPKCSEncryptCb)(SilcBool success,
                                     const unsigned char *encrypted,
                                     SilcUInt32 encrypted_len,
                                     void *context);

   /* Decryption callback */
   typedef void (*SilcPKCSDecryptCb)(SilcBool success,
                                     const unsigned char *decrypted,
                                     SilcUInt32 decrypted_len,
                                     void *context);

   Either add new _async functions or add the callbacks to existing API
   and if the callback is NULL then the API is not async and if provided
   it may be async.  For example;

   SilcBool silc_pkcs_sign(SilcPrivateKey private_key,
                           unsigned char *src, SilcUInt32 src_len,
                           unsigned char *dst, SilcUInt32 dst_size,
                           SilcUInt32 *dst_len,
                           SilcBool compute_hash, SilcHash hash,
                           SilcPKCSSignCb async_sign,
                           void *async_sign_context);

   (if this is done then there's no reason why the buffers in the
    callbacks cannot be the ones user gives here) or allow only async:

   SilcBool silc_pkcs_sign(SilcPrivateKey private_key,
                           unsigned char *src, SilcUInt32 src_len,
                           SilcBool compute_hash, SilcHash hash,
                           SilcPKCSSignCb async_sign,
                           void *async_sign_context);

   or add new:

   SilcBool silc_pkcs_sign_async(SilcPrivateKey private_key,
                                 unsigned char *src, SilcUInt32 src_len,
                                 SilcBool compute_hash, SilcHash hash,
                                 SilcPKCSSignCb async_sign,
                                 void *async_sign_context);

 o Change PKCS Algorithm API to take SilcPKCSAlgorithm as argument to
   encrypt, decrypt, sign and verify functions.  We may need to for exmaple
   check the alg->hash, supported hash functions.  Maybe deliver it also
   to all other functions in SilcPKCSAlgorithm to be consistent.

 o Add DSS support.  Take implementation from Tom or make it yourself.

 o Implement the defined SilcDH API.  The definition is in
   lib/silccrypt/silcdh.h.  Make sure it is asynchronous so that it can
   be accelerated.  Also take into account that it could use elliptic
   curves.

 o ECDSA and ECDH

 o All cipher, hash, hmac etc. allocation routines should take their name
   in as const char * not const unsigned char *.


SILC Accelerator Library
========================

 o SILC Accelerator API.  Provides generic way to use different kind of
   accelerators.  Basically implements SILC PKCS API so that SilcPublicKey
   and SilcPrivateKey can be used but they call the accelerators.

   Something in the lines of (preliminary):

   /* Register accelerator to system.  Initializes the accelerator. */
      Varargs are optional accelerator specific init parameteres. */
   SilcBool silc_acc_register(SilcAccelerator acc, ...);

     silc_acc_register(softacc, "min_threads", 2, "max_threads", 16, NULL);

   /* Unregister accelerator.  Uninitializes the accelerator. */
   SilcBool silc_acc_unregister(const SilcAccelerator acc);

   /* Return list of the registered accelerators */
   SilcDList silc_acc_get_supported(void);

   /* Find existing accelerator.  `name' is accelerator's name. */
   SilcAccelerator silc_acc_find(const char *name);

   /* Return accelerator's name */
   const char *silc_acc_get_name(SilcAccelerator acc);

   /* Accelerate `public_key'.  Return accelerated public key. */
   SilcPublicKey silc_acc_public_key(SilcAccelerator acc,
                                     SilcPublicKey public_key);

   /* Accelerate `private_key'.  Returns accelerated private key. */
   SilcPrivateKey silc_acc_private_key(SilcAccelerator acc,
                                       SilcPrivateKey private_key);

   /* Return the underlaying public key */
   SilcPublicKey silc_acc_get_public_key(SilcAccelerator acc,
                                         SilcPublicKey public_key);

   /* Return the underlaying private key */
   SilcPrivateKey silc_acc_get_private_key(SilcAccelerator acc,
                                           SilcPrivateKey private_key);

   typedef struct SilcAcceleratorObject {
     const char *name;                  /* Accelerator's name */
     SilcBool (*init)(va_list va);      /* Initialize accelerator */
     SilcBool (*uninit)(void);          /* Uninitialize accelerator */
     const SilcPKCSAlgorithm *pkcs;     /* Accelerated PKCS algorithms */
     const SilcDHObject *dh;            /* Accelerated Diffie-Hellmans */
     const SilcCipherObject *cipher;    /* Accelerated ciphers */
     const SilcHashObject *hash;        /* Accelerated hashes */
     const SilcHmacObject *hmac;        /* Accelerated HMACs */
     const SilcRngObject *rng;          /* Accelerated RNG's */
   } *SilcAccelerator, SilcAcceleratorStruct;

   Allows accelerator to have multiple accelerators (cipher, hash etc)
   and multiple different algorithms and implementations (SHA-1, SHA-256 etc).

   SilcPublicKey->SilcSILCPublicKey->RsaPublicKey accelerated as:
   SilcPublicKey->SilcAcceleratorPublicKey->SilcSoftAccPublicKey->
     SilcPublicKey->SilcSILCPublicKey->RsaPublicKey

   silc_acc_public_key creates SilcPublicKey and SilcAcceleratorPublicKey
   and acc->pkcs->import_public_key creates SilcSoftAccPublicKey.

 o Implement software accelerator.  It is a thread pool system where the
   public key and private key operations are executed in threads.

   const struct SilcAcceleratorObject softacc =
   {
     "softacc", softacc_init, softacc_uninit,
     softacc_pkcs, NULL, NULL, NULL, NULL
   }

   /* Called from silc_acc_private_key */
   int silc_softacc_import_private_key(void *key, SilcUInt32 key_len,
                                       void **ret_private_key)
   {
     SilcSoftAccPrivateKey prv = silc_calloc(1, sizeof(*prv));
     prv->pkcs = acc->pkcs;
     prv->private_key = key;
     *ret_private_key = prv;
   }

 (o Symmetric key cryptosystem acceleration?  They are always sycnhronouos
   even with hardware acceleration so the crypto API shouldn't require
   changes.) maybe


lib/silcmath
============

 o Import TFM.  Talk to Tom to add the missing functions.  Use TFM in
   client and client library, but TMA in server, due to the significantly
   increased memory consumption with TFM, and the rare need for public
   key operations in server.

   We want TFM's speed but not TFM's memory requirements.  Talk to Tom
   about making the TFM mp dynamic just as it is in LTM.

 o The SILC MP API function must start returning indication of success
   and failure of the operation.

 o Do SilcStack support for silc_mp_init, silc_mp_init_size and other
   any other MP function (including utility ones) that may allocate
   memory.

 o All utility functions should be made non-allocating ones.


SILC XML Library, lib/silcxml/
==============================

 o SILC XML API (wrapper to expat).  Look at the expat API and simplify
   it.  The SILC XML API should have at most 8-10 API functions.  It should
   be possible to create full XML parser with only one function.  And, it
   should be possible to have a function that is able to parse an entire
   XML document.  It should also have a parser function to be able to
   parse a stream of XML data (SilcStream).  It MUST NOT have operations
   that require multiple function calls to be able to execute that one
   operation (like creating parser).


lib/silcske/silcske.[ch]
========================

 o Ratelimit to UDP/IP transport for incoming packets.


lib/silcasn1
============

 o Negative integer encoding is missing, add it.

 o SILC_ASN1_CHOICE should perhaps return an index what choice in the
   choice list was found.  Currently it is left for caller to figure out
   which choice was found.

 o SILC_ASN1_NULL in decoding should return SilcBool whether or not
   the NULL was present.  It's important when it's SILC_ASN1_OPTIONAL
   and we need to know whether it was present or not.


lib/silcpgp
===========

 o OpenPGP certificate support, allowing the use of PGP public keys
   in SILC.


lib/silcssh
===========

 o SSH2 public key/private key support, allowing the use of SSH2 keys
   in SILC.  RFC 4716.


lib/silcpkix
============

 o PKIX implementation


apps/silcd
==========

 o Deprecate the old server.  Write interface for the new lib/silcserver
   server library.  The interface should work on Unix/Linux systems.

 o Consider deprecating also the old config file format and use XML
   istead.  This should require SILC XML API implementation first.

 o The configuration must support dynamic router and server connections.
   The silcd must work without specifying any servers or routers to
   connect to.

 o The configuration must support specifying whether the server is
   SILC Server or SILC Router.  This should not be deduced from the
   configuration as it was in < 1.2.

 o The configuration must support specifying the ciphers and hmacs and
   their order so that user can specify which algorithms take preference.


lib/silcserver
==============

 o Rewrite the entire server.  Deprecate apps/silcd as the main server
   implementation and create lib/silcserver/.  It is a platform
   independent server library.  The apps/silcd will merely provide a
   a simple interface for the library.

 o Write the SILC Server library extensively using SILC FSM.

 o Server library must support multiple networks.  This means that one
   server must be able to create multiple connections that each reach
   different SILC network.  This means also that all cache's etc. must
   be either connection-specific or network-specific.

 o Library must support dynamic router and server connections.  This means
   that connections are create only when they are needed, like when someone
   says JOIN foo@foo.bar.com or WHOIS foobar@silcnet.org.

 o Library must support server-to-server connections even though protocol
   prohibits that.  The responder of the connection should automatically
   act as a router.  The two servers create an own, isolated, SILC network.
   To be used specifically with dynamic connections.

 o Library must support multiple threads and must be entirely thread safe.

 o Library must have support for SERVICE command.

 o The server must be able to run behind NAT device.  This means that
   Server ID must be based on public IP instead of private IP.

 o The following data must be in per-connection context: client id cache,
   server id cache, channel id cache, all statistics must be
   per-connection.

 o The following data must be in per-thread context: command context
   freelist/pool, pending commands, random number generator.

 o Do inccoming packet processing in an own FSM thread in the
   server-threads FSM.  Same as in client library.

 o Reference count all Silc*Entry structures.

 Some issues that must be kept in mind from 1.0 and 1.1 silcd's:

 o The SERVER_SIGNOFF notify handing is not optimal, because it'll
   cause sending of multiple SIGNOFF notify's instead of the one
   SERVER_SIGNOFF notify that the server received.  This should be
   optimized so that the only SERVER_SIGNOFF is sent and not
   SIGNOFF of notify at all (using SIGNOFF takes the idea about
   SERVER_SIGNOFF away entirely).

 o Another SERVER_SIGNOFF opt/bugfix:  Currently the signoff is
   sent to a client if it is on same channel as the client that
   signoffed.  However, the entire SERVER_SIGNOFF list is sent to
   the client, ie. it may receive clients that was not on the
   same channel.  This is actually against the specs.  It must be
   done per channel.  It shouldn't receive the whole list just
   because one client happened to be on same channel.

 o If client's public key is saved in the server (and doing public key
   authentication) then the hostname and the username information could
   be taken from the public key.  Should be a configuration option!

 o Add a timeout to handling incoming JOIN commands.  It should be
   enforced that JOIN command is executed only once in a second or two
   seconds.  Now it is possible to accept n incoming JOIN commands
   and process them without any timeouts.  THis must be employed because
   each JOIN command will create and distribute the new channel key
   to everybody on the channel.

 o Related to above.  If multiple JOINs are received in sequence perhaps
   new key should be created only once, if the JOINs are handeled at the same
   time.  Now we create multiple keys and never end up using them because
   many JOINs are processed at the same time in sequence.  Only the last
   key ends up being used.