WIKI.org
# NOTE: This file holds additional documentation which doesn't need to
# be in the main README which also gets displayed on Github.
* FAQ
:PROPERTIES:
:CUSTOM_ID: faq
:END:
** Synchronization back-ends
:PROPERTIES:
:CUSTOM_ID: sync_backends
:END:
*** Dropbox
:PROPERTIES:
:CUSTOM_ID: faq_dropbox
:END:
**** Permissions ("Full Dropbox" vs. "App folder")
The reference organice host [[https://organice.200ok.ch/]] has the
permission type "[[https://www.dropbox.com/developers/reference/developer-guide][Full Dropbox]]". This serves many users well -
especially those who have Org files in various locations that they
want to access.
If you're a user and have concerns to give an application full access
to your Dropbox, here are your options:
1. You might not need to change anything after reading this: organice
is a front-end application, there is no back-end and no monitoring
whatsoever. So, when you login to Dropbox, only your browser will
have access to your Dropbox. So, from a security perspective,
you're not giving too much access to a server - your data cannot be
seen by anyone else but you.
2. organice is a free and open source application. Therefore, you can
review the synchronization code anytime. For Dropbox, this is
pretty straight forward since the synchronization code is less than
200loc.
3. If you still have concerns, you can take full control! Since
organice is free and open source, you are free to host it yourself.
You can also create your own integration with Dropbox and select
the permissions as you wish. Here's [[#deployment][more documentation on deploying
organice]].
*** WebDAV
:PROPERTIES:
:CUSTOM_ID: faq_webdav
:END:
**** Demo
Here's a demo of how organice works when logging in to a WebDAV
server.
On the left, you see a branded version of OwnCloud, on the right you
see organice. After logging in and making a minute change, you can see
that the =last edited= timestamp in OwnCloud changes. Also, we're
verifying the change directly in Emacs using Emacs and you can see the
change has also been synchronized to my local machine.
[[https://github.com/200ok-ch/organice/wiki/videos/demo-webdav.gif]]
**** WebDAV test server
:PROPERTIES:
:CUSTOM_ID: webdav_faq_test_server
:END:
For testing purposes, we use a Docker image with a proven and well
documented server: Apache2 running on Debian. You can build the Docker
image yourself - and customize the Apache configuration to your needs:
#+BEGIN_SRC shell
docker build -f doc/webdav/Dockerfile -t apache-webdav .
#+END_SRC
or with docker-compose:
#+begin_src shell
docker-compose build apache-webdav
#+end_src
Then run the Apache2 WebDAV server with:
#+BEGIN_SRC shell
docker run -dit --name apache-webdav-app -p 8080:80 apache-webdav
#+END_SRC
or with docker-compose:
#+begin_src shell
docker-compose up apache-webdav
#+end_src
On your host machine, you can now login with any WebDAV client using
the URL =http://localhost:8080/webdav=. There is no authentication
configuration, so any user account works (including omitted user
accounts). It goes without saying that if you wanted to use this for
production, please enable authentication. Within the test image,
you'll find the [[https://github.com/200ok-ch/organice/blob/master/sample.org][sample.org]] file, so you can get started developing and
testing right away.
For testing WebDAV outside of organice, and you're an Emacs user, you
can open [[/dav:localhost#8080:/webdav/][this]] link (=C-c C-o=). Then, you will get a [[https://www.gnu.org/software/tramp/][TRAMP]] session
with [[https://www.gnu.org/software/emacs/manual/html_node/emacs/Dired.html][dired]] open and you'll see the sample file. You can interact with
it like in any other dired buffer. Obviously, this link will not work
when looking at the documentation in the browser, you'll have to open
the file [[https://github.com/200ok-ch/organice/blob/master/WIKI.org][WIKI.org]] in Emacs.
If you prefer a command line client, you could use [[https://linux.die.net/man/1/cadaver][cadaver]]. Install and use
it like this:
#+BEGIN_SRC shell
sudo apt -y install cadaver
cadaver http://localhost:8080/webdav/
#+END_SRC
***** Bug reports
:PROPERTIES:
:CUSTOM_ID: webdav_bug_reports
:END:
If you have any trouble connecting to WebDAV using organice, it could
be your setup (please consult [[#webdav_cors][CORS]] and [[#webdav_gotchas][Gotchas with WebDAV]]). In any
case, if you want to open a bug report, please document your issue by
referencing how it doesn't work using the official WebDAV test server.
**** CORS
:PROPERTIES:
:CUSTOM_ID: webdav_cors
:END:
Since organice is a front-end application, it will login with
JavaScript from within the browser - in turn the [[https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS][Cross-Origin Resource
Sharing (CORS)]] headers must be set appropriately. If they are not set,
you will not be able to login to your service from a browser.
Alternatively, if you're using a server like Apache or Nginx, you can
simply get around CORS by hosting organice on the same domain as your
service.
Please note, that when your back-end does not set the correct CORS
headers, organice cannot show you a really semantic error message on
that. The reason is that browsers [[https://www.w3.org/TR/cors/#handling-a-response-to-a-cross-origin-request][hide this information]] from
JavaScript. You will simply get a network error. However, you can
easily debug it yourself by looking into the JavaScript console. No
worries, you don't have to be a (JavaScript) developer to find out
about that - here's a screencast showing you how to do it:
[[https://github.com/200ok-ch/organice/wiki/videos/demo-webdav-failing-cors.gif]]
**** Gotchas with WebDAV
:PROPERTIES:
:CUSTOM_ID: webdav_gotchas
:END:
***** =preflight request doesn't pass access control check= error
If you are getting an error like
#+BEGIN_EXAMPLE
Access to XMLHttpRequest at 'https://my.site/dav/' from origin
'https://organice.200ok.ch' has been blocked by CORS policy: Response
to preflight request doesn't pass access control check: It does not
have HTTP ok status.
#+END_EXAMPLE
then something is wrong with your webserver config. You can check
whether a CORS preflight check is returning the right headers via:
#+BEGIN_EXAMPLE
curl -v -X OPTIONS https://my.server/webdav/
# For the official organice Apache2 WebDAV test server:
# curl -v -X OPTIONS http://localhost:8080/webdav/
#+END_EXAMPLE
The output should include lines like this:
#+BEGIN_EXAMPLE
...
< HTTP/1.1 200 OK
...
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET,POST,OPTIONS,DELETE,PUT,PROPFIND
< Access-Control-Allow-Headers: Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth
< Access-Control-Allow-Credentials: true
< Allow: OPTIONS,GET,HEAD,POST,DELETE,TRACE,PROPFIND,PROPPATCH,COPY,MOVE,LOCK,UNLOCK
#+END_EXAMPLE
If your server doesn't give a =200 OK= response, or if the
=Access-Control-Allow-*= headers are missing, you may find these
articles helpful:
- https://stackoverflow.com/questions/27703871/return-empty-response-from-apache/
- https://serverfault.com/questions/231766/returning-200-ok-in-apache-on-http-options-requests/
***** Using Apache =RewriteEngine=
:PROPERTIES:
:CUSTOM_ID: webdav_apache_rewrite_engine
:END:
If your WebDAV directory happens to be not only on the same webserver,
but also within a subdirectory of the directory containing a
=.htaccess= file containing a =RewriteRule= that also applies to the
WebDAV directory (for example like [[#routing][this]]), then you will need to create
another =.htaccess= file in the top-level WebDAV directory containing
this:
#+BEGIN_EXAMPLE
RewriteEngine Off
#+END_EXAMPLE
Otherwise any attempts to use WebDAV to upload new files via =HTTP
PUT= requests will fall foul of the =/index.html= rewrite rule above,
resulting in a =403 Forbidden= response.
Another way to avoid this more selectively is to precede that rule
with:
#+BEGIN_EXAMPLE
RewriteCond %{REQUEST_METHOD} !PUT
#+END_EXAMPLE
***** Symlinks don't work
Unfortunately, [[https://serverfault.com/questions/453807/best-practice-to-link-with-webdav-as-followsymlinks-doesn-t-allow-to-show-symli][symlink support never made it into Apache's =mod_dav=]].
***** Bind-mounts of individual files don't work
In an Apache =mod_dav= context, unfortunately you can't use [[https://unix.stackexchange.com/questions/198590/what-is-a-bind-mount][bind
mounts]] of a single file instead of symlinks, because =mod_dav=
attempts to write any changes to a file atomically, by first writing
to a temporary file and then [[https://github.com/apache/httpd/blob/c3db73ca8a5aa7b79231a11fe2eb15de3ce943dc/modules/dav/fs/repos.c#L991][atomically renaming it to the target
file]], and Linux prevents renaming to bind mounts with a =Device or
resource busy= error.
***** =HTTP PUT= requests fail with =403 Forbidden=
As mentioned in [[#routing][the section routing]], you should avoid having
=mod_rewrite= rules apply to (=PUT=) requests in the WebDAV
directories.
**** Configuring Nextcloud behind haproxy to allow WebDAV
If you're running Nextcloud behind [[https://www.haproxy.com/][haproxy]] it's entirely possible to use it with
organice using WebDAV. ...it's just a little bit convoluted.
The first part is the haproxy config. It should look a little bit like this:
#+NAME: /etc/haproxy/haproxy.conf
#+BEGIN_SRC conf
frontend www
acl host_nextcloud hdr(host) nextcloud.example.org
acl path_nextcloud_public_webdav path_beg /public.php/webdav
# Because we need to inspect the path in the backend section we set a variable
# containing the path.
http-request set-var(txn.path) path
# Because the OPTIONS requests from organice doesn't include authentication we
# need to fake it. We can do that by redirecting all requests that satisfy these conditions:
#
# + host is Nextcloud
# + path is for public webdav
# + HTTP method is OPTIONS
use_backend always200ok if host_nextcloud path_nextcloud_public_webdav METH_OPTIONS
# haproxy doesn't really have a way of returning an arbitrary response, unless
# you want to drop down to Lua. There's no need for that, though, as this works
# perfectly fine. This backend doesn't have any servers attached, so it'll
# always result in a 503. We override the 503 by setting a custom errorfile,
# which incidentally looks just like an HTTP 200 response and contains all the
# headers we need to satisfy a CORS request.
backend always200ok
mode http
errorfile 503 /etc/haproxy/errors/200-ok.http
# The Nextcloud server backend is configured here. We inject CORS headers if URL
# starts with `/public.php/webdav`.
backend nextcloud
mode http
option httplog
acl is_webdav var(txn.path) -m beg /public.php/webdav
http-response add-header Access-Control-Allow-Origin "*" if is_webdav
http-response add-header Access-Control-Allow-Methods "GET,POST,OPTIONS,DELETE,PUT,PROPFIND" if is_webdav
http-response add-header Access-Control-Allow-Headers "Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth" if is_webdav
http-response add-header Access-Control-Allow-Credentials "true" if is_webdav
server backend01 127.0.0.1:8001
#+END_SRC
The ~errorfile~ needs to look something like the below. Note that the text below
has carriage returns (~13~, ~o15~ or ~0x0d~); these are required as per the HTTP
RFC!
#+NAME: /etc/haproxy/errors/200-ok.http
#+BEGIN_SRC text
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: close
Content-Type: text/html
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,POST,OPTIONS,DELETE,PUT,PROPFIND
Access-Control-Allow-Headers: Origin,Accept,X-Requested-With,Content-Type,Access-Control-Request-Method,Access-Control-Request-Headers,Authorization,X-CSRF-Token,Depth
Access-Control-Allow-Credentials: true
<html><body><h1>200 Stuff is good!</h1>
Something something dark side.
</body></html>
#+END_SRC
**** Nextcloud sharing
In order to share a document using WebDAV you might be inclined to try to follow
the [[https://docs.nextcloud.com/server/16/user_manual/files/access_webdav.html#accessing-public-shares-over-webdav][official documentation]], but it can be a tad confusing. Here's the executive
summary for how to share things from Nextcloud using WebDAV:
+ share a link to a folder/file
+ remove everything but the token from the link; the token matches
~/[a-zA-Z0-9]+$/~ (hit the button right of "Share link" if using the web
interface)
+ use these details when logging in:
+ URL :: https://nextcloud.example.org/public.php/webdav
+ Username :: the token, e.g. ~ed65Fxw9Bz3MTn3~
+ Password :: if you've set a password for the shared folder, here's where you
input it
**** WebDAV Config
:PROPERTIES:
:CUSTOM_ID: webdav-config
:END:
If you want to self-host organice and want to configure a default
WebDAV server for your instance, you can by setting the variable
=REACT_APP_WEBDAV_URL= in the =.env= file.
** Can we add feature X from plugin Y?
:PROPERTIES:
:CUSTOM_ID: can-we-add-feature-x-from-plugin-y
:END:
organice is an implementation of [[http://orgmode.org/][Org mode]] (see [[#what-does-this-project-do][What does this
project do?]]). Therefore, it is important that the changes in the
markup made by organice are 100% compatible with Org mode itself.
Hence, if feature X from plugin Y can be implemented in a
compatible way, and the feature follows the [[#contributing][contribution guideline]],
then: Yes, the feature can be added to organice.
* Development
** Architecture Decision Records
:PROPERTIES:
:CUSTOM_ID: architecture-decision-records
:END:
*** adr-000-template
:PROPERTIES:
:CUSTOM_ID: adr-template
:END:
#+BEGIN_SRC org
,* TITLE <short present tense imperative phrase, less than 50 characters, like a git commit message.>
,** Status
<proposed, accepted, rejected, deprecated, superseded, etc.>
,** Context
# <what is the issue that we're seeing that is motivating this decision
# or change.>
,** Decision
# <what is the change that we're actually proposing or doing.>
,** Consequences
# <what becomes easier or more difficult to do because of this change.>
#+END_SRC
*** adr-001-use-adrs
:PROPERTIES:
:CUSTOM_ID: adr-001
:END:
**** Architecture Decision Record: Use ADRs
***** Context
organice aims to be a helpful and accessible tool for many users
and developers over years to come. Hence, practicing discipline of
architecture is very important.
- We want to think deeply about all our architectural decisions,
exploring all alternatives and making a careful, considered,
well-researched choice.
- Even when the above statement does not hold true and we walk a
pragmatic path, we want to be as transparent as possible in our
decision-making process.
- We want to be able to revisit prior decisions to determine fairly if
they still make sense, and if the motivating circumstances or
conditions have changed.
***** Decision
We will document every architecture-level decision for organice
with an [[http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions][Architecture Decision Record]]. These are a well structured,
relatively lightweight way to capture architectural proposals. They
can serve as an artifact for discussion, and remain as an enduring
record of the context and motivation of past decisions.
The workflow will be:
1. A developer creates an ADR document outlining an approach for a
particular question or problem. The ADR has an initial status of
"proposed."
2. The developers and maintainers discuss the ADR. During this
period, the ADR should be updated to reflect additional context,
concerns raised, and proposed changes.
3. Once a maintainer has made a decision, the ADR can be
transitioned to either an "accepted" or "rejected" state.
4. Since the team working on organice is very agile and good architecture
often emerges from actual code and spikes, it is very well possible
for code to already be committed to the repository before the ADR
is accepted or even addressed as an ADR.
5. If a decision is revisited and a different conclusion is reached, a
new ADR should be created documenting the context and rationale for
the change. The new ADR should reference the old one, and once the
new one is accepted, the old one should (in its "status" section)
be updated to point to the new one. The old ADR should not be
removed or otherwise modified except for the annotation pointing to
the new ADR.
We will use the popular ADR template by [[https://github.com/joelparkerhenderson/architecture-decision-record/blob/4fba870a4e8e6830d0b9aa150c145103aa34b634/templates/decision-record-template-by-michael-nygard/index.md][Michal Nygard]] using [[#adr-001][this
template]].
***** Status
Accepted
***** Consequences
1. Developers must write an ADR and submit it for review to make
any architectural decision transparent -- that is, any decision
that affects the way organice is put together at a high level.
2. We will have a concrete artifact around which to focus discussion,
before finalizing decisions.
3. If we follow the process, decisions will be made deliberately,
by the maintainers.
4. We will have a useful persistent record of why the system is the way it is.
*** adr-002-static-content-pages
:PROPERTIES:
:CUSTOM_ID: adr-002
:END:
**** Architecture Decision Record: Static Content Pages
***** Context
organice runs different kinds of pages:
1. Static content like the Landing Page
2. The actual application
These two do not share a whole lot in common. They have different
needs in terms of CSS, Javascript, but also code quality. However,
they run from the same =<App>= component which already has the
assumption that it'll contain the actual application, not a content
page like the Landing Page.
This might seem convoluted and maybe there is a better solution. The
seemingly only other solution is to =yarn eject= from =react-scripts=,
so that we could have multiple starting HTML files for multiple SPAs.
This has the following downsides, though:
1. =react-scripts= is a great wrapper which we cannot use anymore.
2. Related to 1: Ejecting will create dozens of files which will have
to be maintained manually.
***** Decision
The downsides of ejecting seem larger than sharing the same =<App>=
component and creating safeguards in the code where necessary to
differentiate between the two use cases.
***** Status
Accepted
***** Consequences
- Some safeguards in the code like:
- Setting a =landing-page= class dynamically in =<Entry>= if the
application is showing the Landing Page.
- Making sure that static content pages do not =import= CSS code
that pollutes the global namespace (reminder: CSS is not component
local by default in React). Instead, we employ SCSS and scope all
selectors under a unique identifier.
*** adr-003-synchronization-back-ends
:PROPERTIES:
:CUSTOM_ID: adr-003
:END:
**** Synchronization back-ends
***** Context
Different users like to synchronize their files using different
back-ends. Some prefer open standards and to host the required services
themselves - others prefer to pay for a closed source service so that
they don't have to worry about storage of their data themselves. The
spread of strategies employed by these back-ends couldn't be further
apart - some work in hunks and commits, some on files, etc.
To accommodate every personal synchronization strategy, organice
should be easily extensible.
***** Decision
organice employs the [[https://en.wikipedia.org/wiki/Strategy_pattern][Strategy Pattern]] which is a [[https://en.wikipedia.org/wiki/Software_design_pattern][design pattern]] made
to do the 'same thing' (like storing and retrieving files) with
'different strategies' (like employing different APIs for Dropbox,
GitLab, and WebDAV).
***** Status
Accepted
***** Consequences
Implementing a client for a new synchronization back-end is quite
easy:
1. Take any of the [[https://github.com/200ok-ch/organice/tree/master/src/sync_backend_clients][existing back-ends]] as a template. If you already
happen to know how the API of your new synchronization back-end
works, you can pick the closest one. Ultimately, it doesn't make a
big difference from where you start off.
2. Implement the [[https://github.com/200ok-ch/organice/blob/d00e1e31b3c675b3be71c0e0150f8ad7292766b0/src/sync_backend_clients/webdav_sync_backend_client.js#L133-L143][8 functions defined by the strategy pattern]]:
1. =isSignedIn=
2. =getDirectoryListing=
3. =getMoreDirectoryListing=
4. =updateFile=
5. =createFile=
6. =getFileContentsAndMetadata=
7. =getFileContents=
8. =deleteFile=
3. That's it😂
* Building this documentation
:PROPERTIES:
:CUSTOM_ID: building_docs
:END:
This comprehensive documentation is an aggregation of multiple files
which all reside in the [[https://github.com/200ok-ch/organice][organice code repository]] (=README.org=,
=WIKI.org=, =CONTRIBUTING.org=, and =CODE_OF_CONDUCT.md=).
To build the documentation locally, run =make docs=.
Building the documentation and uploading it to
https://organice.200ok.ch/documentation.html is part of the [[https://github.com/200ok-ch/organice/blob/master/.circleci/config.yml][CI/CD
workflow]]. The actual compilation happens [[https://github.com/200ok-ch/organice/blob/master/bin/compile_doc.sh][here]] and the result gets
uploaded [[https://github.com/200ok-ch/organice/blob/master/bin/compile_doc_and_upload.sh][here]].