Fix crate-level docs not being rendered correctly if there is no readme

[GuillaumeGomez]

Fixes #2725.

This is an interesting case since there is no README, so instead we tool the crate level docs. But we did a few things wrong:

• Markdown was not rendered
• Highlighting wasn't run as expected (it was never picking rust)
• Lines start were completely trimmed whereas they shouldn't

All fixed in this PR.

@syphar Is there a way to test this except for GUI tests?

[syphar]

just some questions for my understanding:

We're (kind of) replicating things that rustdoc already did when generating the docs themselves.

Isn't there any way that we could either reuse the machinery? or is rustdoc so near to normal markdown that we're fine here?

[GuillaumeGomez]

just some questions for my understanding:

We're (kind of) replicating things that rustdoc already did when generating the docs themselves.

Isn't there any way that we could either reuse the machinery? or is rustdoc so near to normal markdown that we're fine here?

Apart from intra-doc links and headings limitations, there is nothing special in rustdoc markdown handling. Also, it's all internal rustdoc code, so unless we write a compiler plugin, we can't access to it. And I don't think it's worth it.

[GuillaumeGomez]

I made the default syntax configurable and optional. Now, if there is a README, it will have the same behaviour as now. However, if it's coming from the crate-level docs because there is no README, it will apply the same rules as rustdoc: if there is no language specified, then by default it's rust.

[syphar]

@syphar Is there a way to test this except for GUI tests?

@GuillaumeGomez the test you're probably searching for is this one:

docs.rs/src/web/crate_details.rs

Lines 2119 to 2188 in d97b27d

	fn readme() {
	async_wrapper(|env| async move {
	env.fake_release()
	.await
	.name("dummy")
	.version("0.1.0")
	.readme_only_database("database readme")
	.create()
	.await?;
	env.fake_release()
	.await
	.name("dummy")
	.version("0.2.0")
	.readme_only_database("database readme")
	.source_file("README.md", b"storage readme")
	.create()
	.await?;
	env.fake_release()
	.await
	.name("dummy")
	.version("0.3.0")
	.source_file("README.md", b"storage readme")
	.create()
	.await?;
	env.fake_release()
	.await
	.name("dummy")
	.version("0.4.0")
	.readme_only_database("database readme")
	.source_file("MEREAD", b"storage meread")
	.source_file("Cargo.toml", br#"package.readme = "MEREAD""#)
	.create()
	.await?;
	env.fake_release()
	.await
	.name("dummy")
	.version("0.5.0")
	.readme_only_database("database readme")
	.source_file("README.md", b"storage readme")
	.no_cargo_toml()
	.create()
	.await?;
	let check_readme = |path: String, content: String| {
	let env = env.clone();
	async move {
	let resp = env.web_app().await.get(&path).await.unwrap();
	let body = resp.text().await.unwrap();
	assert!(body.contains(&content));
	}
	};
	check_readme("/crate/dummy/0.1.0".into(), "database readme".into()).await;
	check_readme("/crate/dummy/0.2.0".into(), "storage readme".into()).await;
	check_readme("/crate/dummy/0.3.0".into(), "storage readme".into()).await;
	check_readme("/crate/dummy/0.4.0".into(), "storage meread".into()).await;
	let mut conn = env.async_db().await.async_conn().await;
	let details = crate_details(&mut conn, "dummy", "0.5.0", None).await;
	assert!(matches!(
	details.fetch_readme(&*env.async_storage().await).await,
	Ok(None)
	));
	Ok(())
	});
	}

You could create a release without readme, but with description_long, that validates if syntax highlighting is added

[GuillaumeGomez]

Added the regression test. :)