YXORP

YXORP is a reverse proxy for the HTTP protocol. The main focus for YXORP is checking the validity of the traffic, i.e. what would be called 'application layer firewalling' or 'secure reverse proxy' in commercial products. It can check, validate, and change just about anything about a HTTP request or response, can load balance, and all other normal reversed proxy functions. YXORP aims to conform to RFC 2616, RFC 2518, RFC 2109 and other applicable standards.

What YXORP does is receive requests, break all fields and headers apart and feed the information in them to a script engine. The scripts can change all fields except for a few that are required for the connections to work correctly; for instance the Connection: header and the Content-Length header. After the scripts have finished, YXORP then sends the request - including any changes that have been made - on to the web server.

YXORP is written in plain C; it should run on any Unix variant as long as it supports reasonably recent standards. It requires a couple of libraries to build, like libxml2, openssl, and pcre - all mainstream Unix versions have these by default or allow the required development libraries to be easily installed.

There used to be two sets of releases - development and stable. Since the amount of development is rather low in recent years - because YXORP is a mature and stable software - I've decided to no longer differentiate between these. Especially since most major sites that run YXORP that I'm aware of use the 'development' releases anyway. In case of bugs, the last two releases will be supported; bug fixes may still be produced for older versions but this will be decided on a case by case basis.

Changelog for release 2.34 (latest):

• fix for #3586630, not processing the Connection: header correctly in the presence of token values other than close.
• fix for issue when serving chunked, then unchunked content from a cache - the content-length was not correctly set.

Changelog for release 2.33:

• changed processing of basicauth with regard to ldap. Previously, using basic_auth_check would cause yxorp to read the clear text password from the ldap. Now, it uses the credentials in the basic authentication to try a simple bind to the ldap. New attribute to support this is bindfmt on the ldapsource tag.
• basicauth credentials verified with an ldap server are now cached. The time that credentials are cached are set by the cachetime attribute on the ldapsource tag. The table is periodically cleaned up; this is governed by the globalconfig setting basicauthmaintenanceinterval.
• added Via: header processing, including new globalconfig settings viareportversionstringtoclient, viareportpackagenametoclient, viareporthosttoclient, viareportversionstringtoserver, viareportpackagenametoserver, viareporthosttoserver. Default is to include a full entry into an existing Via: header, or create a new one for both requests and responses.
• fixed bugs dealing with setting the xlateid attribute in customized header definitions; a) the default setting for xlateid was incorrect; b) configured setting of xlateid was not always completely copied into the actual configuration.
• issue warning when configure has not detected a usable ldap library
• include ldap support in showbuild and daemon output
• changed default for options method to allow outbound entities. The options method is still disabled by default [thanks to Rene Klootwijk]
• removed all ldap code related to ldapsource and ldapserver tags
• started new ldap implementation based on rule functions; functions added are ldap_set_serveruri, ldap_set_searchdn, ldap_set_searchpw, ldap_set_searchscope, ldap_search, ldap_set_searchbase, ldap_init, ldap_bind. Also, changed the interface for checking basic authentication; old function basic_auth_check was removed; functions added are basicauth_add, basicauth_check, basicauth_getuser, basicauth_getpass, basicauth_reject for processing basic authentication. Effectively, with this new interface, the 'old' technology of setting basic authentication strings in the configuration is still available, and is now augmented by a very flexible interface to ldap; the ldap query results can be cached in the basic authentication table for optimization.
• fix for #3024257, new globalconfig items maxchunkheaderlength and maxchunkheaderreadretries.
• fix for bug in the rewrite of location: headers which could cause malformed headers and/or duplicate headers [thanks to Rene Klootwijk for identifying this issue]

Changelog for release 2.32:

• client states can now be mapped by including function calls in a rule that derive some information from the request that is unique enough to reliably map into a state entry. An example of such information might be a jsession cookie that an application server has generated, or some unique bits in the URL. The new functions are setclientstateid() and setclientstateidgenerate().
• entries where the URI has parameters are no longer added to the cache.
• the generated configuration, ie. from yxorpconfig -r, now includes a read-only attribute that shows whether or not a listener has opened the socket it is associated with. The attribute is named open.
• added a new section of information-only output to generated configuration ie. yxorpconfig -r. The tag name of this section is daemon.
• added comment tags to the various sections in the generated configuration.
• mkinstallconfig script now also installs the dtd.
• header Authentication-Info: added to the default table.
• header Cookie2: added to the default header table.
• build problem in ypriv fixed [thanks to Ferenc].
• mode attribute in virtualserver added to dtd.
• adding support for sni, rfc4366/3546. New tag to be included inside listener tag is certlist, takes attributes of hostname, certfile, certpasswd, cafile.
• certificates defined on a listener (in sni context, these are the default certificates that a non-sni capable browser will get) can now be updated.
• changed default header buffer check for Server: header not to reject, because Apache2's default may send out a Server: header that violates RFC2616.
• many changes to the sources to build without warnings, in spite of a higher default warning level.
• added globalconfiguration item localhostname, this allows to set which name will be reported in for instance Via: and X-Cache: headers.
• added X-Cache: header to header table.
• added globalconfiguration item generatexcache to control whether X-Cache headers are generated. Note that X-Cache headers are only generated if caching is applicable to the request; if the request is not cachable, no header is generated irrespective of the setting of generatexcache.
• added weighted random scheduling, virtualserver attribute schedule value "weightedrandom". Also, weight tag on virtualserver real tag.
• removed mode parameter from virtualserver, replaced by clearrealservers tag.
• added add and remove tags to virtualserver real tag.
• added virtualserver command.
• several changes to fix a limitation in the maximum content length. Previously, this was limited at 2Gb; now, it is unlimited.
• fixed a memory leak occurring when server side ssl was used.
• added globalconfiguration item workerthreadstacksize to enable tuning of memory mapping in case yxorp is running many threads. In this scenario, most platform defaults for the thread stack size are much larger than yxorp requires, and issues with system-imposed limits may arise. Preferably, though, platform tools (like ulimit) should be used; this configuration item exists only where using the platform tool is not practicable.
• the default error log fmt no longer contains %clientiplookup.
• feature request 1860197: attribute added on listener tag, pvtkeyfile, enables to load private key from a separate file instead of the same file as the certfile. If this attribute is not set, yxorp will try to load the private key from the certfile.
• feature request 1860197: attribute added on certlist tag, pvtkeyfile, enables to load private key from a separate file instead of the same file as the certfile. If this attribute is not set, yxorp will try to load the private key from the certfile.
• ssl items in a listener could be initialized twice [thanks to Rene Klootwijk].
• fixed bug in message formatting when running on a system without functioning name resolver [thanks to Rene Klootwijk].
• added checks if sufficient entropy can be found for use by OpenSSL. If not, yxorp will refuse to start [thanks to Rene Klootwijk].
• feature request 1860197: yxorp no longer checks where the configuration files reside (including ssl certs, keys, etc).
• Please note when upgrading from previous versions: feature request 1860195: yxorp by default now installs binaries for both the daemon and tools in /.../sbin. When upgrading, please remove the old version before running 'make install', and check your init scripts for the correct path.
• added new function strremove to rule language.
• added clientprotocol and serverprotocol as formats for request and error logs. Default for both logs is now serverprotocol.
• changed processing of wakeup for out-of-service real servers. If all defined servers are out of service, always try to wake one up.
• Location: headers that contain a hostname (in absoluteURI form) are now rewritten to the hostname that the client originally used. This can be disabled by the new setting rewritelocationheader in globalconfig.
• changed default check for content-type header to text.
• added option renameonrotate in globalconfiguration if 1, yxorp will append a time stamp to the old (rotated) log file after closing it. If 0, which is default, yxorp will just close and then reopen the log files.
• added functions setsslserverconnecttimeout, setsslserverreadtimeout, setsslserverwritetimeout, setsslserverclosetimeout, setconnectservertimeout, setreadfromservertimeout setwritetoservertimeout; these can be used to modify the timeout values used for the server connection for specific requests.

Changelog for release 2.31:

• make sure bytes in threadlog-hexdump are not sign extended. Same for debugging hexdump.
• retry connecting to server if the initial read from the server returns 0 ie. session shutdown.
• yxorpconfig -rv now only shows enabled headers. Same for methods.
• report the serverconnectionretrycounter in rdlog internals output.
• added folding attribute into header table. At this time, it will not yet be enabled for any headers besides XFF.
• added apparently forgotten locking in count_yxorp_threads()
• changed defaults for methods OPTIONS, PUT, DELETE; these are now by default enabled if the methodgroups=rfc2616 attribute is enabled in the globalconfiguration.
• changed default for header DAV buffercheck, is now rfc2616_text.
• changed processing of inbound entities without Content-Length: header. If the method carries the dav or msext groups, yxorp will assume that no inbound entity is present if no Content-Length: header is present on the request.
• added function killclientstate(), this will immediately remove the entire client state associated with the current request. Synonym is clearclientstate().
• added function killsticky(), synonym clearsticky() which clears the sticky mapping for the current request.
• added function setclientstatefastage() to set a faster age timer on an individual client state entry. This is to be used by a rule in the situation that a session is ending, but some requests may still be coming in that we will want to handle on the original state.
• changed the behaviour of the stickyloss and sorry rules. These rule types no longer run as reject rules and implicitly end the request. Sorry rules may now be used to cause a retry to another server (including scheduling to another virtualserver), and or with changed request fields. Special variables connectionretries and maxconnectionretries are set in the sorryrule context to allow sorryrules to check whether retries are exhausted. Stickyloss rules may either cause a redirect, or an explicit reject. It is also possible to forward the request to another server group, even though this probably will not make sense, since no session context will be available on this other server. In the case of a stickylost rule, the special variable stickylost is set to a value of 1. Both for the stickylost and the sorry rules, using a reject in a rule will cause the defined reject rule to run before the end of the request processing.
• added option -l to yxorp command to load configuration from the file that was set on the command that originally started the daemon.
• Also, option to the yxorp command -l can be set as --load, and -V as --version.
• Several options (-d, -D, -v, -X) now cause an error if yxorp was built without debugging.
• Added code to more predictably detect if the session from yxorp to a backend server is available for reuse.
• Increased default maximum for the thread pool to 200; increased the minimum free to 6; increased initial to 20. Since 2.26 threads exit after a configurable time of inactivity, so the increased numbers will settle at minimum amounts for an idle yxorp after some time, while this may prevent problems for heavy users. Also added a new keyword startatoverrun that governs how many threads are started if minfree is reached.
• done some updates to make parsing and tools code more maintainable.