"RST" and "ReST" (or "reST") are both acceptable. Care should be taken with capitalization, to avoid confusion with "REST", an acronym for "Representational State Transfer".
The abbreviations "reSTX" and "rSTX"/"rstx" should not be used; they overemphasize reStructuredText's precedessor, Zope's StructuredText.