YXORP

YXORP is a reverse proxy for the HTTP protocol. The main focus for YXORP is checking the validity of the traffic, i.e. what would be called 'application layer firewalling' or 'secure reverse proxy' in commercial products. It can check, validate, and change just about anything about a HTTP request or response, can load balance, and all other normal reversed proxy functions. YXORP aims to conform to RFC 2616, RFC 2518, RFC 2109 and other applicable standards.

A stable release is available.

YXORP is developed on Linux (both i386 and x86_64) and Solaris (Sparc and i386/x86_64), and should also run on *BSD variants including Mac OS X.

Latest stable version

Download the latest version stable-1.23.9 of YXORP.
Bugs in the latest stable version may be reported here.

Bugs fixed since last released version:

• 1929367 : header attribute xlatename not processed correctly.

Known bugs: See the current list here.

Feature status

• SSL support (including backend SSL) -- OK
• Full support for the HTTP 1.1 protocol -- OK
• On-line configuration and reconfiguration using XML configuration files -- OK
• Virtual hosting: several web sites can be handled by a single reverse proxy -- OK
• Rule based processing of requests: all requests can be allowed, rejected, or changed depending on rules -- OK
• Multiple listeners: one YXORP can monitor several TCP ports and process incoming traffic -- OK
• Load balancing over multiple servers, including automatic setting inservice and outservice, and redirection to "sorry" server pages if no server is available -- OK
• Sticky load balancing -- OK
• Use pcre instead of glibc regex -- OK

Latest development release

Download the latest development version 2.33 of YXORP.

Planned major features for 2nd stable release:

• Caching to enable reducing load on the target server (completed)
• Pattern scanning of HTTP entities (partially complete, might be taken out of 2.xx release schedule)
• Authorization of web resources (basic auth completed, digest authentication disabled for now)
• WebDAV support (completed)

Changelog for development release 2.33 (latest):

• changed processing of basicauth with regard to ldap. Previously, using basic_auth_check would cause yxorp to read the clear text password from the ldap. Now, it uses the credentials in the basic authentication to try a simple bind to the ldap. New attribute to support this is bindfmt on the ldapsource tag.
• basicauth credentials verified with an ldap server are now cached. The time that credentials are cached are set by the cachetime attribute on the ldapsource tag. The table is periodically cleaned up; this is governed by the globalconfig setting basicauthmaintenanceinterval.
• added Via: header processing, including new globalconfig settings viareportversionstringtoclient, viareportpackagenametoclient, viareporthosttoclient, viareportversionstringtoserver, viareportpackagenametoserver, viareporthosttoserver. Default is to include a full entry into an existing Via: header, or create a new one for both requests and responses.
• fixed bugs dealing with setting the xlateid attribute in customized header definitions; a) the default setting for xlateid was incorrect; b) configured setting of xlateid was not always completely copied into the actual configuration.
• issue warning when configure has not detected a usable ldap library
• include ldap support in showbuild and daemon output
• changed default for options method to allow outbound entities. The options method is still disabled by default [thanks to Rene Klootwijk]
• removed all ldap code related to ldapsource and ldapserver tags
• started new ldap implementation based on rule functions; functions added are ldap_set_serveruri, ldap_set_searchdn, ldap_set_searchpw, ldap_set_searchscope, ldap_search, ldap_set_searchbase, ldap_init, ldap_bind. Also, changed the interface for checking basic authentication; old function basic_auth_check was removed; functions added are basicauth_add, basicauth_check, basicauth_getuser, basicauth_getpass, basicauth_reject for processing basic authentication. Effectively, with this new interface, the 'old' technology of setting basic authentication strings in the configuration is still available, and is now augmented by a very flexible interface to ldap; the ldap query results can be cached in the basic authentication table for optimization.
• fix for #3024257, new globalconfig items maxchunkheaderlength and maxchunkheaderreadretries.
• fix for bug in the rewrite of location: headers which could cause malformed headers and/or duplicate headers [thanks to Rene Klootwijk for identifying this issue]

Changelog for development release 2.32:

• client states can now be mapped by including function calls in a rule that derive some information from the request that is unique enough to reliably map into a state entry. An example of such information might be a jsession cookie that an application server has generated, or some unique bits in the URL. The new functions are setclientstateid() and setclientstateidgenerate().
• entries where the URI has parameters are no longer added to the cache.
• the generated configuration, ie. from yxorpconfig -r, now includes a read-only attribute that shows whether or not a listener has opened the socket it is associated with. The attribute is named open.
• added a new section of information-only output to generated configuration ie. yxorpconfig -r. The tag name of this section is daemon.
• added comment tags to the various sections in the generated configuration.
• mkinstallconfig script now also installs the dtd.
• header Authentication-Info: added to the default table.
• header Cookie2: added to the default header table.
• build problem in ypriv fixed [thanks to Ferenc].
• mode attribute in virtualserver added to dtd.
• adding support for sni, rfc4366/3546. New tag to be included inside listener tag is certlist, takes attributes of hostname, certfile, certpasswd, cafile.
• certificates defined on a listener (in sni context, these are the default certificates that a non-sni capable browser will get) can now be updated.
• changed default header buffer check for Server: header not to reject, because Apache2's default may send out a Server: header that violates RFC2616.
• many changes to the sources to build without warnings, in spite of a higher default warning level.
• added globalconfiguration item localhostname, this allows to set which name will be reported in for instance Via: and X-Cache: headers.
• added X-Cache: header to header table.
• added globalconfiguration item generatexcache to control whether X-Cache headers are generated. Note that X-Cache headers are only generated if caching is applicable to the request; if the request is not cachable, no header is generated irrespective of the setting of generatexcache.
• added weighted random scheduling, virtualserver attribute schedule value "weightedrandom". Also, weight tag on virtualserver real tag.
• removed mode parameter from virtualserver, replaced by clearrealservers tag.
• added add and remove tags to virtualserver real tag.
• added virtualserver command.
• several changes to fix a limitation in the maximum content length. Previously, this was limited at 2Gb; now, it is unlimited.
• fixed a memory leak occurring when server side ssl was used.
• added globalconfiguration item workerthreadstacksize to enable tuning of memory mapping in case yxorp is running many threads. In this scenario, most platform defaults for the thread stack size are much larger than yxorp requires, and issues with system-imposed limits may arise. Preferably, though, platform tools (like ulimit) should be used; this configuration item exists only where using the platform tool is not practicable.
• the default error log fmt no longer contains %clientiplookup.
• feature request 1860197: attribute added on listener tag, pvtkeyfile, enables to load private key from a separate file instead of the same file as the certfile. If this attribute is not set, yxorp will try to load the private key from the certfile.
• feature request 1860197: attribute added on certlist tag, pvtkeyfile, enables to load private key from a separate file instead of the same file as the certfile. If this attribute is not set, yxorp will try to load the private key from the certfile.
• ssl items in a listener could be initialized twice [thanks to Rene Klootwijk].
• fixed bug in message formatting when running on a system without functioning name resolver [thanks to Rene Klootwijk].
• added checks if sufficient entropy can be found for use by OpenSSL. If not, yxorp will refuse to start [thanks to Rene Klootwijk].
• feature request 1860197: yxorp no longer checks where the configuration files reside (including ssl certs, keys, etc).
• Please note when upgrading from previous versions: feature request 1860195: yxorp by default now installs binaries for both the daemon and tools in /.../sbin. When upgrading, please remove the old version before running 'make install', and check your init scripts for the correct path.
• added new function strremove to rule language.
• added clientprotocol and serverprotocol as formats for request and error logs. Default for both logs is now serverprotocol.
• changed processing of wakeup for out-of-service real servers. If all defined servers are out of service, always try to wake one up.
• Location: headers that contain a hostname (in absoluteURI form) are now rewritten to the hostname that the client originally used. This can be disabled by the new setting rewritelocationheader in globalconfig.
• changed default check for content-type header to text.
• added option renameonrotate in globalconfiguration if 1, yxorp will append a time stamp to the old (rotated) log file after closing it. If 0, which is default, yxorp will just close and then reopen the log files.
• added functions setsslserverconnecttimeout, setsslserverreadtimeout, setsslserverwritetimeout, setsslserverclosetimeout, setconnectservertimeout, setreadfromservertimeout setwritetoservertimeout; these can be used to modify the timeout values used for the server connection for specific requests.

Changelog for development release 2.31:

• make sure bytes in threadlog-hexdump are not sign extended. Same for debugging hexdump.
• retry connecting to server if the initial read from the server returns 0 ie. session shutdown.
• yxorpconfig -rv now only shows enabled headers. Same for methods.
• report the serverconnectionretrycounter in rdlog internals output.
• added folding attribute into header table. At this time, it will not yet be enabled for any headers besides XFF.
• added apparently forgotten locking in count_yxorp_threads()
• changed defaults for methods OPTIONS, PUT, DELETE; these are now by default enabled if the methodgroups=rfc2616 attribute is enabled in the globalconfiguration.
• changed default for header DAV buffercheck, is now rfc2616_text.
• changed processing of inbound entities without Content-Length: header. If the method carries the dav or msext groups, yxorp will assume that no inbound entity is present if no Content-Length: header is present on the request.
• added function killclientstate(), this will immediately remove the entire client state associated with the current request. Synonym is clearclientstate().
• added function killsticky(), synonym clearsticky() which clears the sticky mapping for the current request.
• added function setclientstatefastage() to set a faster age timer on an individual client state entry. This is to be used by a rule in the situation that a session is ending, but some requests may still be coming in that we will want to handle on the original state.
• changed the behaviour of the stickyloss and sorry rules. These rule types no longer run as reject rules and implicitly end the request. Sorry rules may now be used to cause a retry to another server (including scheduling to another virtualserver), and or with changed request fields. Special variables connectionretries and maxconnectionretries are set in the sorryrule context to allow sorryrules to check whether retries are exhausted. Stickyloss rules may either cause a redirect, or an explicit reject. It is also possible to forward the request to another server group, even though this probably will not make sense, since no session context will be available on this other server. In the case of a stickylost rule, the special variable stickylost is set to a value of 1. Both for the stickylost and the sorry rules, using a reject in a rule will cause the defined reject rule to run before the end of the request processing.
• added option -l to yxorp command to load configuration from the file that was set on the command that originally started the daemon.
• Also, option to the yxorp command -l can be set as --load, and -V as --version.
• Several options (-d, -D, -v, -X) now cause an error if yxorp was built without debugging.
• Added code to more predictably detect if the session from yxorp to a backend server is available for reuse.
• Increased default maximum for the thread pool to 200; increased the minimum free to 6; increased initial to 20. Since 2.26 threads exit after a configurable time of inactivity, so the increased numbers will settle at minimum amounts for an idle yxorp after some time, while this may prevent problems for heavy users. Also added a new keyword startatoverrun that governs how many threads are started if minfree is reached.
• done some updates to make parsing and tools code more maintainable.

Changelog for development release 2.30:

• corrected error in the reading of <digestauth> tags in the configuration.
• include <!DOCTYPE> tag in the configuration.
• added DTD for the configuration XML.
• added attribute clearfilters to debug tag in config.
• fixed incorrect setting of default portnumber on realserver tag. This caused problems if using a non-default portnumber while using the yxorprealserver command to change realserver status.

Changelog for development release 2.29:

• bug fixed in parsing <requestdetaillog> configuration.
• bug fixed in processing requests; a buffer was not cleared potentially leading to including data from another request. This bug was introduced in version 2.27, beta-testers of 2.27 or 2.28 are strongly recommended to upgrade.
• bug fixed in processing chunked data; the symptom is that the request after the request with chunked data processing is rejected with 'Server response does not contain HTTP-Version'
• added bitmask 0x02 to rdlog - causes hexdump of message data in each of the receive/transfer stages.
• added bitmask 0x04 to rdlog - causes dump of internal data structures in each of the receive/transfer stages.

Changelog for development release 2.28:

• changed wrong interpretation of XFF in the situation where multiple proxies exist.
• no longer generate debug messages for client states that are not eligible for timeout during cleanup runs.
• added new type of log file: the requestdetail log file. Set <log requestdetail=... /> to use it.
• allow online reconfiguration to turn debugging flags off. Module filters still can not be reconfigured.
• added internal request data to rdlog
• added header set-cookie2
• added received, transmitted request and reply to rdlog
• added functions rdlog, rdlogerror, rdlogreceivedrequest, rdlogtransmittedrequest, rdlogreceivedresponse, rdlogtransmittedresponse, rdlogfinalinternaldata to manipulate rdlog settings for individual requests

Changelog for development release 2.27:

• added globalconfiguration items doclientheaderenabledchecks, doclientheaderlengthchecks, doclientheaderbufchecks, doclientheaderduplicatechecks, doserverheaderenabledchecks, doserverheaderlengthchecks, doserverheaderbufchecks, doserverheaderduplicatechecks. The purpose of this is to enable to switch these checks of globally, instead of having to modify the entire header table. This may be useful for people that run yxorp as a load balancer, as opposed to a reverse proxy for security purposes. Please note that disabling these checks (default is on) may expose your servers to security attacks. Also please note that disabling these checks does not mean that Yxorp will never reject a request. Many more checks on requests and responses are done; for some, enable/disable switches may appear in future versions, but not for all, since Yxorp needs to be able to understand enough of a request to be able to process it.
• changed generation of yxorpconfig -r output; for globalconfiguration values, only those changed from default will be reported.
• changed generation of yxorpconfig -r output; for headers, only those changed from default will be reported.
• changed generation of yxorpconfig -r output; for status codes, only those changed from default will be reported.
• increased maxlen for User-Agent: and Pragma:.
• removed hard-coded value for MAX_DISSECTBUF, which was the maximum size of a request from a client (excl. http-entity). Now defined in globalconfiguration item workdatadissectbufsize. Same for MAX_RESPONSEBUF.
• removed hard-coded value for MAX_WBUF, which was an internal buffer required to be larger than MAX_DISSECTBUF. Now defined in globalconfiguration item workdatadissectwbufsize.
• removed hard-coded MAXLEN_REQUESTURI which limited the maximum length of an URI that yxorp could process. Now defined in globalconfiguration item maxlenrequesturi.
• removed hard-coded MAXLEN_HEADERLINE which was the maximum length of any header that yxorp could process. Now defined in globalconfiguration item maxlenheaderline.
• headers for which the maximum length depends on the maximum allowed length of an URI can now specify a maximum length relative to the maximum URI length, by setting a negative number. The absolute of this number will be added to the max URI length.
• doclientheaderunknownchecks, doserverheaderunknownchecks to govern whether or not unknown headers (ie. not in yxorp's table of known headers) will be passed through. No other checks will be performed on these, regardless of other settings.
• changed startup message in case not started by root.
• added basic log rotation using a HUP signal or -R command flag to yxorp.
• improved checking for overlength header: state length of offending header in message, also check for \r presence separately.
• default max length for From: header increased.
• don't check for header name to be followed by SP, since rfc2616 does not demand this. Also accept HT to follow header name.
• changed most header default length checks to ignore instead of reject.
• fixed compile problem in ypriv.c that occurred when compiling without security model.
• fixed issue in mkinstallconfig script, in which if the default sysconfdir directory was changed, the directory was not created if it did not exist. Also fixed several minor issues with installing certificates. [thanks to Ferenc Born]
• improved checking for libcap.

Changelog for development release 2.26:

• source of request.c split up to improve maintainability
• more correctly report status codes for 405, 408, 414, 501 and 505 cases. Also, the default of returning 400 whatever problem is encountered is now no longer default; it can however still be enforced by using a reject rule.
• statecookiename in globalconfig, value of (auto) which is default sets the cookie name to a derivative of the hostname. This overcomes problems that otherwise occur with two chained yxorp servers.
• setstatecookiename() function
• digest authentication method added. This comprises the digest_auth_check function, and the <digestauth> tag in the configuration.
• (experimental) support for ldap added, tags <ldapserver> and <ldapsource>
• added informational/ro attributes actual, actualfree to thread tag in generated config
• minor bug fixing in reading the host: header; port 0 is not accepted any more.
• settargetserver() function added; %targetserver added to log formatting; somewhat more clear distinction in messages and yxorpclientstate output on whether hosts or servers are intended; checked that sticky scheduling is based on host (set by rule or read from header) rather than server. Rules for sticky scheduling are obtained from server definitions, however.
• improved checking on mutexes (cache.c, clientstate.c, entbufmm.c, ldapsource.c, ldapserver.c, requestcount.c, ssldlock.c, ssllock.c)
• rules may now contain no code
• added refreshage to <cachearea> tag; this is used in conjunction with maxage, to control refreshing of a cached object before it expires. This may be used to refresh the cache under high load, while avoiding to stress the target server. Default is one quarter of maxage.
• modified ipv6/ipv4 code in listener.c, retrieve actual listener address used on connection
• retrieve actual address used by yxorp on connection to server
• added functions setsendxforwardedfor() and sanitizexforwardedfor()
• added functions sanitizexforwardedforstrict() and sanitizexforwardedforloose(). -strict demands that the client address matches the last entry in an X-Forwarded-For: header. Normal demands that the client address matches one of the ranges that are passed. -loose does not look at the client address at all.
• added Content-Disposition: header, enabled by default
• added counting of client state mappings to real servers, generated if -vv on yxorprealserver
• added setclientipfromxff() function
• changed X-Forwarded-For to be enabled by default in the header table. Note that yxorp will not by default process, or send, an xff to the server however; this still needs calls to setsendxforwardedfor and sanitizexforwardedfor**.
• added (experimental) ldap support for digest authentication
• added generic feature to threading system: thread cleanup callbacks. Code can now register a callback that will be invoked when the tread exits.
• threads now exit after a configurable time of inactivity; the time is set using threadinactivitylimit in globalconfig. The <thread minfree="x" /> config item determines how many threads will remain active.
• changed default bufcheck for Authorization: header to rfc2616_text because Authorization carries a copy of the uri when digest_auth is used
• changed default timers for ssl to server
• improved error logging, bug caused hostname not to be included if a request was rejected in the dissect cycle
• increased size of clientstatetable hash
• solved timing problem in cache aging
• stabilized ldap code

Changelog for development release 2.25:

• ssl client certificates are now supported between the client and yxorp. New configuration items are (in listener) requestclientcertificate, verifyclientcertificate; (in globalconfig) sslclientaccepttimeout, sslclientreadtimeout, sslclientwritetimeout, sslclientclosetimeout. Session timers are now also active for ssl client sessions.
• certificate generation is completely redone, the sample configuration can now generate a certificate authority and use that to sign server and client certificates.
• ssl serverend is now nonblocking and obeys timer settings, new globalconfig items sslserverconnecttimeout, sslserverreadtimeout, sslserverwritetimeout, sslserverclosetimeout.
• added functions isccertverified(), isscertverified(), getclientcipherbits(), getservercipherbits(), getclientciphername(), getserverciphername(), getclientcertfailcode(), getservercertfailcode(), getclientcertfailmsg(), getservercertfailmsg(), getclientcertinfo(), getclientcertissuerinfo(), getservercertinfo(), getservercertissuerinfo()
• changed the way that dvar's are freed; this no longer frees an entry in the table, but just invalidates it. This improves performance and makes debugging output clearer.
• added function killvariable()
• added realserver configuration items verifyservercertificate, cafile to support checking of server certificates
• added setssclientcertificate()
• changed default for POST, it will now by default also process outbound entities
• fixed a bug in memory allocation for dvar tables
• added id callback function for openssl, added support for openssl dynamic locks
• orderly shutdown; globalconfig.clearthreadpoolgracetime, globalconfig.closelogdaemongracetime
• corrected error message for sessions in which the ssl handshake was not correctly completed
• installing sample configuration redone to satisfy make distcheck
• check if configuration is viable before starting daemon
• globalconfig.starterwaitforsignaltime

Changelog for development release 2.24:

• added switch -v to the yxorp command to add timestamps to debugging output messages. This makes it easier to link debugging output to other log messages.
• the user and group attributes in the security configuration can now be names instead of numbers. Names are used in the generated configuration as well.
• removed the (default 4K) size limitation for dvar objects and objects used in rules. There still is a maximum size, but this is very large and configurable in the globalconfig.
• there is now a unified way of checking for headers that illegally occur more than once. This is governed by a new attribute "duplicatesallowed" in the globalconfig header table. The technique has been seen in the wild as 'content-length piggybacking' but can theoretically be applied to other headers as well. Yxorp can now check for this for all headers.
• new rule types added "inboundentity" and "outboundentity". These are set on the listener. Also, these types can be used on the definitions of the rules.
• added methods to global configuration.
• added getrequestnumber() function
• added getentity() function
• the table of status codes can now be configured in the global configuration
• added numeric buffer checking table
• added method and header definitions used by Microsoft implementations of WebDAV
• added functions to query and change which rules run after the request stage
• obsoleted findheader function
• the reject attribute in the header table has been replaced by a more fine grained mechanism, where each of the header attributes can selectively be used to allow a header, or cause the header to be ignored, or the complete request or response to be rejected.
• added hop-by-hop attribute to header table. This is mainly for internal use by Yxorp, and should normally not be changed.
• added some logic to omit defaults from generated header configurations
• added cachearea attribute to listener
• getcachearea, setcachearea functions added
• added hdrgroup header attribute, possible values are currently "dav" and "msext". Also added globalconfig item enabledheadergroups
• added methodgroup method attribute, possible values are "rfc2616", "dav", "msext". Globalconfig item that globally enables methodgroups is called enabledmethodgroups.
• added cachemaintenanceinterval globalconfig item
• added entitybuffermaxsize globalconfig item
• added logic to suppress defaults from generated configurations for the globalconfig, header, and reason codes sections
• improvements in the handling of incomplete requests. Sessions that are closed due to a mismatch in keep-alive settings no longer cause half-filled log entries, and also no longer cause reject rules to run.
• added cache configuration element, with attribute maxage that causes the cache entries to expire after a configurable time
• the cache can now serve requests with the HEAD method, and GET with if-modified-since. Also, the cache now obeys client side cache control directives
• added maxobjectsize attribute to cachearea
• responses which carry a vary: header are not added to the cache - #1590816: when ./configure is run several times, using different settings of --prefix or --sysconfdir, the scripts that install the sample configuration and sample ssl certificate get installed in the sysconfdir location that was set on the first configure run [thanks to Ferenc Born]
• changed the way that yxorp starts. The default is now that yxorp itself arranges to run as a daemon, instead of relying on external tools to accomplish this, because these tools are not available on all platforms [thanks to Ferenc Born for pointing this out] New flags to the yxorp command have been added: -N to force running in the foreground, and -K to stop a running daemon. Besides these changes, yxorp now creates a pid file, default location is /var/run/yxorp/yxorp.pid (this may be changed in the globalconfiguration).
• added request pipelining, including correct handling of inbound fixed-length and chunked content, and setting the received data bytes for logging correctly
• set handler for sigpipe in mainline, not in a worker thread
• run wthread maintenance processes faster than once a second
• changed numerous reject message texts
• changed processing of inbound (both server and client) data where the received length is zero
• split up the request.c module to improve code management
• changed SSL processing to non-blocking, so the timeout values in the globalconfig are obeyed