Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(dw-edit-managed-sources): Edit Self Managed Sources for DW #28111

Open
wants to merge 21 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
6cf654a
Setting up a new page for managing a self-managed source
phixMe Jan 29, 2025
f9a5ef4
Form is submitting.
phixMe Jan 30, 2025
6fb3d6d
Form resetting and resolving form issues.
phixMe Jan 30, 2025
3e7eada
Fixing error, removing some unused code, and improving readability.
phixMe Jan 30, 2025
1113676
Unused code removal and cache invalidation.
phixMe Jan 30, 2025
e2d2455
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Jan 30, 2025
dd5ee1a
Auto code review updates.
phixMe Jan 30, 2025
95c6862
Removing keys from update for now.
phixMe Jan 31, 2025
e33e6ff
Override update method.
phixMe Jan 31, 2025
c366540
Code review comments.
phixMe Feb 3, 2025
d2f8058
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
6886dd8
Update UI snapshots for `chromium` (2)
github-actions[bot] Feb 3, 2025
07e9e32
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
56a4b03
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 3, 2025
59511d8
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 4, 2025
7d8a233
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 5, 2025
d9a9849
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 5, 2025
cd97dbe
Kea logic colliding remediation
phixMe Feb 6, 2025
24d8e2c
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
f3e075c
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
68b1925
Merge branch 'master' into feature(dw-edit-managed-sources)
phixMe Feb 6, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 80 additions & 49 deletions frontend/src/scenes/data-warehouse/new/DataWarehouseTableForm.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { LemonInput, LemonSelect, Link } from '@posthog/lemon-ui'
import { LemonButton, LemonInput, LemonSelect, Link } from '@posthog/lemon-ui'
import { useValues } from 'kea'
import { Form } from 'kea-forms'
import { LemonField } from 'lib/lemon-ui/LemonField'
Expand Down Expand Up @@ -44,7 +44,11 @@ const ProviderMappings: Record<
},
}

export function DatawarehouseTableForm(): JSX.Element {
interface Props {
onUpdate?: () => void
}

export function DatawarehouseTableForm({ onUpdate }: Props): JSX.Element {
const { manualLinkingProvider } = useValues(sourceWizardLogic)

const provider = manualLinkingProvider ?? 'aws'
Expand All @@ -59,65 +63,85 @@ export function DatawarehouseTableForm(): JSX.Element {
>
<div className="flex flex-col gap-2 max-w-160">
<LemonField name="name" label="Table name">
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder="Examples: stripe_invoice, hubspot_contacts, users"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder="Examples: stripe_invoice, hubspot_contacts, users"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<div className="text-muted text-xs mb-4">This will be the table name used when writing queries</div>
<LemonField name="url_pattern" label="Files URL pattern">
<LemonInput
data-attr="table-name"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].fileUrlPatternPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="table-url-pattern"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].fileUrlPatternPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<div className="text-muted text-xs mb-4">
You can use <strong>*</strong> to select multiple files.
</div>
<LemonField name="format" label="File format" className="w-max mb-4">
<LemonSelect
data-attr="table-format"
options={[
{ label: 'Parquet (recommended)', value: 'Parquet' },
{ label: 'CSV', value: 'CSV' },
{ label: 'CSV with headers', value: 'CSVWithNames' },
{ label: 'JSON', value: 'JSONEachRow' },
{ label: 'Delta', value: 'Delta' },
]}
/>
{({ value = '', onChange }) => (
<LemonSelect
data-attr="table-format"
options={[
{ label: 'Parquet (recommended)', value: 'Parquet' },
{ label: 'CSV', value: 'CSV' },
{ label: 'CSV with headers', value: 'CSVWithNames' },
{ label: 'JSON', value: 'JSONEachRow' },
{ label: 'Delta', value: 'Delta' },
]}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<LemonField name={['credential', 'access_key']} label={ProviderMappings[provider].accessKeyLabel}>
<LemonInput
data-attr="access-key"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].accessKeyPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="access-key"
className="ph-ignore-input"
placeholder={ProviderMappings[provider].accessKeyPlaceholder}
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
<LemonField name={['credential', 'access_secret']} label={ProviderMappings[provider].accessSecretLabel}>
<LemonInput
data-attr="access-secret"
className="ph-ignore-input"
type="password"
placeholder="eg: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
/>
{({ value = '', onChange }) => (
<LemonInput
data-attr="access-secret"
className="ph-ignore-input"
type="password"
placeholder="eg: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
autoComplete="off"
autoCapitalize="off"
autoCorrect="off"
spellCheck={false}
value={value}
onChange={onChange}
/>
)}
</LemonField>
{provider === 'google-cloud' && (
<div className="text-muted text-xs">
Expand All @@ -127,6 +151,13 @@ export function DatawarehouseTableForm(): JSX.Element {
</Link>
</div>
)}
{!!onUpdate && (
<div className="flex justify-end">
<LemonButton type="primary" onClick={onUpdate}>
Update
</LemonButton>
</div>
phixMe marked this conversation as resolved.
Show resolved Hide resolved
)}
</div>
</Form>
)
Expand Down
83 changes: 33 additions & 50 deletions frontend/src/scenes/data-warehouse/new/NewSourceWizard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { PageHeader } from 'lib/components/PageHeader'
import { FEATURE_FLAGS } from 'lib/constants'
import { featureFlagLogic } from 'lib/logic/featureFlagLogic'
import { useCallback } from 'react'
import { DataWarehouseSourceIcon } from 'scenes/data-warehouse/settings/DataWarehouseSourceIcon'
import { SceneExport } from 'scenes/sceneTypes'

import { ManualLinkSourceType, SourceConfig } from '~/types'
Expand All @@ -13,7 +14,6 @@ import SchemaForm from '../external/forms/SchemaForm'
import SourceForm from '../external/forms/SourceForm'
import { SyncProgressStep } from '../external/forms/SyncProgressStep'
import { DatawarehouseTableForm } from '../new/DataWarehouseTableForm'
import { RenderDataWarehouseSourceIcon } from '../settings/DataWarehouseManagedSourcesTable'
import { dataWarehouseTableLogic } from './dataWarehouseTableLogic'
import { sourceWizardLogic } from './sourceWizardLogic'

Expand Down Expand Up @@ -52,17 +52,8 @@ interface NewSourcesWizardProps {
export function NewSourcesWizard({ onComplete }: NewSourcesWizardProps): JSX.Element {
const wizardLogic = sourceWizardLogic({ onComplete })

const {
modalTitle,
modalCaption,
isWrapped,
currentStep,
isLoading,
canGoBack,
canGoNext,
nextButtonText,
showSkipButton,
} = useValues(wizardLogic)
const { modalTitle, modalCaption, isWrapped, currentStep, isLoading, canGoBack, canGoNext, nextButtonText } =
useValues(wizardLogic)
const { onBack, onSubmit } = useActions(wizardLogic)
const { tableLoading: manualLinkIsLoading } = useValues(dataWarehouseTableLogic)

Expand Down Expand Up @@ -96,7 +87,7 @@ export function NewSourcesWizard({ onComplete }: NewSourcesWizardProps): JSX.Ele
</LemonButton>
</div>
)
}, [currentStep, isLoading, manualLinkIsLoading, canGoNext, canGoBack, nextButtonText, showSkipButton])
}, [currentStep, isLoading, manualLinkIsLoading, canGoNext, canGoBack, nextButtonText, onBack, onSubmit])

return (
<>
Expand Down Expand Up @@ -163,33 +154,29 @@ function FirstStep(): JSX.Element {
{
title: 'Source',
width: 0,
render: function RenderAppInfo(_, sourceConfig) {
return <RenderDataWarehouseSourceIcon type={sourceConfig.name} />
render: function (_, sourceConfig) {
return <DataWarehouseSourceIcon type={sourceConfig.name} />
},
},
{
title: 'Name',
key: 'name',
render: function RenderName(_, sourceConfig) {
return (
<span className="font-semibold text-sm gap-1">
{sourceConfig.label ?? sourceConfig.name}
</span>
)
},
render: (_, sourceConfig) => (
<span className="font-semibold text-sm gap-1">
{sourceConfig.label ?? sourceConfig.name}
</span>
),
},
{
key: 'actions',
width: 0,
render: function RenderActions(_, sourceConfig) {
return (
<div className="flex flex-row justify-end">
<LemonButton onClick={() => onClick(sourceConfig)} className="my-2" type="primary">
Link
</LemonButton>
</div>
)
},
render: (_, sourceConfig) => (
<div className="flex flex-row justify-end">
<LemonButton onClick={() => onClick(sourceConfig)} className="my-2" type="primary">
Link
</LemonButton>
</div>
),
},
]}
/>
Expand All @@ -208,33 +195,29 @@ function FirstStep(): JSX.Element {
{
title: 'Source',
width: 0,
render: function RenderAppInfo(_, sourceConfig) {
return <RenderDataWarehouseSourceIcon type={sourceConfig.type} />
},
render: (_, sourceConfig) => <DataWarehouseSourceIcon type={sourceConfig.type} />,
},
{
title: 'Name',
key: 'name',
render: function RenderName(_, sourceConfig) {
return <span className="font-semibold text-sm gap-1">{sourceConfig.name}</span>
},
render: (_, sourceConfig) => (
<span className="font-semibold text-sm gap-1">{sourceConfig.name}</span>
),
},
{
key: 'actions',
width: 0,
render: function RenderActions(_, sourceConfig) {
return (
<div className="flex flex-row justify-end">
<LemonButton
onClick={() => onManualLinkClick(sourceConfig.type)}
className="my-2"
type="primary"
>
Link
</LemonButton>
</div>
)
},
render: (_, sourceConfig) => (
<div className="flex flex-row justify-end">
<LemonButton
onClick={() => onManualLinkClick(sourceConfig.type)}
className="my-2"
type="primary"
>
Link
</LemonButton>
</div>
),
},
]}
/>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ export const dataWarehouseTableLogic = kea<dataWarehouseTableLogicType>([
updateTableSuccess: async ({ table }) => {
lemonToast.success(<>Table {table.name} updated</>)
actions.editingTable(false)
actions.loadDatabase()
router.actions.replace(urls.dataWarehouse())
},
})),
Expand All @@ -92,6 +93,13 @@ export const dataWarehouseTableLogic = kea<dataWarehouseTableLogicType>([
table: {
defaults: { ...NEW_WAREHOUSE_TABLE } as DataWarehouseTable,
errors: ({ name, url_pattern, credential, format }) => {
const HOGQL_TABLE_NAME_REGEX = /^[a-zA-Z_][a-zA-Z0-9_]*$/
phixMe marked this conversation as resolved.
Show resolved Hide resolved
if (!HOGQL_TABLE_NAME_REGEX.test(name)) {
return {
name: 'Invalid table name. Table names must start with a letter or underscore and contain only alphanumeric characters or underscores.',
}
}

if (url_pattern?.startsWith('s3://')) {
return {
url_pattern:
Expand Down
Loading
Loading